{"id":99884,"date":"2020-04-24T10:05:08","date_gmt":"2020-04-24T02:05:08","guid":{"rendered":"http:\/\/4563.org\/?p=99884"},"modified":"2020-04-24T10:05:08","modified_gmt":"2020-04-24T02:05:08","slug":"golang-%e9%80%9a%e8%bf%87-sarama-%e8%bf%9e%e6%8e%a5%e5%88%b0-kafka-%e7%9a%84%e7%96%91%e6%83%91","status":"publish","type":"post","link":"http:\/\/4563.org\/?p=99884","title":{"rendered":"Golang \u901a\u8fc7 sarama \u8fde\u63a5\u5230 kafka \u7684\u7591\u60d1"},"content":{"rendered":"
\n
\n
\n

Golang \u901a\u8fc7 sarama \u8fde\u63a5\u5230 kafka \u7684\u7591\u60d1 <\/h1>\n

<\/p>\n

\n
\u8cc7\u6df1\u5927\u4f6c : hijoker <\/span> <\/i> 50<\/span> <\/div>\n
<\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/p><\/div>\n
<\/p>\n

\/\/ \u95ee\u9898:<\/p>\n

Golang \u901a\u8fc7 sarama \u8fde\u63a5\u5230 kafka,\u6ca1\u6709\u6307\u5b9a group,\u8fde\u63a5\u7684 topic \u53ea\u6709\u4e00\u4e2a partition,\u901a\u8fc7 2 \u53f0\u673a\u5668\u4e0a\u4e00\u6837\u7684\u7a0b\u5e8f\u8fde\u4e0a kafka \u4e0a, \u8fd9 2 \u4e2a\u7a0b\u5e8f\u90fd\u80fd\u6d88\u8d39\u8fd9\u4e00\u4e2a partition \u7684\u6570\u636e.<\/p>\n

\u90a3\u4e48\u8fd9\u4e2a 2 \u53f0\u673a\u5668\u4e0a\u7684\u5e94\u7528\u7a0b\u5e8f\u662f\u5c5e\u4e8e\u540c\u4e00\u4e2a group \u7684,\u800c\u4e14\u662f\u540c\u4e00\u4e2a consumer? \u8981\u77e5\u9053\u4e00\u4e2a partition \u53ea\u80fd\u88ab\u6bcf\u4e2a group \u7684\u4e00\u4e2a consumer \u6d88\u8d39\u7684!!!<\/p>\n

\/\/ kafka \u7248\u672c:<\/p>\n

[email protected]:\/usr\/local\/kafka_2.11-2.0.0\/bin# .\/kafka-topics.sh –version 2.0.0 (Commit:3402a8361b734732)<\/p>\n

\/\/ topic \u53ea\u6709\u4e00\u4e2a partition<\/p>\n

[email protected]:\/usr\/local\/kafka_2.11-2.0.0\/bin# kafka-topics.sh –describe –zookeeper 127.0.0.1:2181 –topic topic.XXX Topic:topic.skygram PartitionCount:1 ReplicationFactor:1 Configs: Topic: topic.XXX Partition: 0 Leader: 0 Replicas: 0 Isr: 0<\/p>\n

\/\/ \u53ea\u6709\u4e00\u4e2a group,\u770b\u4e0d\u5230\u5176\u4ed6\u7684 group \u548c consumer \u7684\u6d88\u606f<\/p>\n

[email protected]:\/usr\/local\/kafka_2.11-2.0.0\/bin# .\/kafka-consumer-groups.sh –bootstrap-server localhost:9092 –list<\/p>\n

KMOffsetCache-ubuntu18<\/p>\n

\/\/ \u786e\u5b9e\u662f\u4e24\u53f0\u673a\u5668\u8fde\u4e0a\u7684<\/p>\n

[email protected]:\/usr\/local\/kafka_2.11-2.0.0\/bin# netstat -antop|grep 9092|grep ESTABLISHED tcp 0 0 192.168.198.145:50396 192.168.198.145:9092 ESTABLISHED 18330\/.\/main_linux keepalive (1.00\/0\/0) tcp 0 0 192.168.198.145:50394 192.168.198.145:9092 ESTABLISHED 18330\/.\/main_linux keepalive (4.07\/0\/0) tcp 0 0 192.168.198.145:50392 192.168.198.145:9092 ESTABLISHED 18330\/.\/main_linux keepalive (4.07\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50394 ESTABLISHED 4367\/java keepalive (3630.28\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54181 ESTABLISHED 4367\/java keepalive (1688.40\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:43006 ESTABLISHED 4367\/java keepalive (4265.90\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54183 ESTABLISHED 4367\/java keepalive (1688.40\/0\/0) tcp6 0 0 192.168.198.145:43012 192.168.198.145:9092 ESTABLISHED 4699\/java keepalive (4266.62\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54182 ESTABLISHED 4367\/java keepalive (1688.40\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50396 ESTABLISHED 4367\/java keepalive (3742.75\/0\/0) tcp6 0 0 192.168.198.145:43006 192.168.198.145:9092 ESTABLISHED 4699\/java keepalive (4265.90\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50392 ESTABLISHED 4367\/java keepalive (3630.28\/0\/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:43012 ESTABLISHED 4367\/java keepalive (4266.62\/0\/0<\/p>\n<\/p><\/div>\n

\u5927\u4f6c\u6709\u8a71\u8aaa<\/b> (6<\/span>) <\/div>\n
<\/div>\n<\/p><\/div>\n<\/p><\/div>\n
    \n
  • \n
    \n
    \n
    \u8cc7\u6df1\u5927\u4f6c : znood <\/span> <\/div>\n
    <\/i> <\/span> <\/div>\n<\/p><\/div>\n
    \u6ca1\u6709\u6307\u5b9a group\uff0c\u90a3 group \u53ef\u80fd\u662f\u968f\u673a\u7684\uff0c\u53bb zookeeper \u91cc\u53ef\u4ee5\u770b\u5230\u6709\u54ea\u4e9b comsumer group <\/div>\n<\/p><\/div>\n<\/li>\n
  • \n
    \n
    \n
    \u4e3b<\/span> \u8cc7\u6df1\u5927\u4f6c : hijoker <\/span> <\/div>\n
    <\/i> <\/span> <\/div>\n<\/p><\/div>\n
    @znood zk \u91cc\u9762\u662f\u7a7a\u7684,\u5f88\u662f\u4ee4\u6211\u56f0\u60d1
    [zk: localhost:2181(CONNECTED) 2] ls \/
    [cluster, controller_epoch, controller, brokers, zookeeper, kafka-manager, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]
    [zk: localhost:2181(CONNECTED) 3] ls \/consumers
    [] <\/div>\n<\/p><\/div>\n<\/li>\n
  • \n
    \n
    \n
    \u4e3b<\/span> \u8cc7\u6df1\u5927\u4f6c : hijoker <\/span> <\/div>\n
    <\/i> <\/span> <\/div>\n<\/p><\/div>\n
    \u6211\u4e5f\u504f\u5411\u8ba4\u4e3a\u8fd9\u662f 2 \u4e2a group,\u4f46\u662f\u600e\u4e48\u4ece kafka \u7684\u540e\u53f0\u6765\u8bc1\u660e\u8fd9\u662f 2 \u4e2a group \u5462
    \u8fd8\u6709,\u597d\u50cf\u65b0\u7248\u672c\u7684 kafka \u7684 consumer \u7684\u4fe1\u606f\u5df2\u7ecf\u4e0d\u5728 zk \u91cc\u4e86
    \u662f\u7528\u8fd9\u4e2a\u547d\u4ee4\u6765\u770b<\/p>\n

    .\/kafka-consumer-groups.sh –bootstrap-server localhost:9092 –list <\/div>\n<\/p><\/div>\n<\/li>\n

  • \n
    \n
    \n
    \u8cc7\u6df1\u5927\u4f6c : znood <\/span> <\/div>\n
    <\/i> <\/span> <\/div>\n<\/p><\/div>\n
    \u90a3 group \u6709\u53ef\u80fd\u662f\u7a7a\u7684\uff0cKMOffsetCache-ubuntu18 \u53ef\u80fd\u662f\u547d\u4ee4\u884c\u6d4b\u8bd5\u751f\u6210\u7684 <\/div>\n<\/p><\/div>\n<\/li>\n
  • \n
    \n
    \n
    \u4e3b<\/span> \u8cc7\u6df1\u5927\u4f6c : hijoker <\/span> <\/div>\n
    <\/i> <\/span> <\/div>\n<\/p><\/div>\n
    @znood https:\/\/www.google.com\/search?biw=1920&bih=937&sxsrf=ACYBGNRoNLV3NleeP-K5T0mwRDkF8D-A6g%3A1579151812697&ei=xPEfXvyKKreu0PEPsda2wAQ&q=KMOffsetCache&oq=KMOffsetCache&gs_l=psy-ab.12..35i39i19j35i39l2j0i10.4516664.4516664..4517922…3.0..2.1015.3690.4-1j1j2j1……0….2j1..gws-wiz…….33i160.1BsHGQK4MGc&ved=0ahUKEwj8_N32rofnAhU3FzQIHTGrDUgQ4dUDCAs
    \u4e0d\u662f\u547d\u4ee4\u884c,\u8fd9\u662f\u7531\u4e8e\u8fd9\u79cd\u6d88\u8d39\u8005\u7684\u4fe1\u606f\u662f\u5b58\u653e\u5728 __consumer_offsets \u4e2d <\/div>\n<\/p><\/div>\n<\/li>\n
  • \n
    \n
    \n
    \u8cc7\u6df1\u5927\u4f6c : znood <\/span> <\/div>\n
    <\/i> <\/span> <\/div>\n<\/p><\/div>\n
    \u4e0d\u7ba1 KMOffsetCache-ubuntu18 \u600e\u4e48\u6765\u7684\uff0c\u6211\u7684\u610f\u601d\u662f\u4f60\u7684\u4e24\u4e2a\u7a0b\u5e8f\u53ef\u80fd\u90fd\u4f7f\u7528\u7684\u7a7a groupID\uff0c\u6211\u641c\u4e86\u4e0b\uff0cgroupID \u53ea\u662f\u4e00\u4e2a label \u6807\u8bc6\uff0c\u9ed8\u8ba4\u4e3a\u7a7a\uff0c\u4e5f\u5c31\u662f\u53ef\u4ee5\u4e3a\u7a7a
    Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.<\/p>\n

    https:\/\/stackoverflow.com\/questions\/43118671\/kafka-consumer-default-group-id<\/p>\n

    https:\/\/cwiki.apache.org\/confluence\/display\/KAFKA\/KIP-289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer <\/p><\/div>\n<\/p><\/div>\n<\/li>\n

  • \n","protected":false},"excerpt":{"rendered":"

    Golang \u901a\u8fc7 sarama …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[],"tags":[],"_links":{"self":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts\/99884"}],"collection":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=99884"}],"version-history":[{"count":0,"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts\/99884\/revisions"}],"wp:attachment":[{"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=99884"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=99884"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=99884"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}