最具影响力的数字化技术在线社区
开启辅助访问 合作方案投稿需求拼课切换到宽版

168大数据

 找回密码
 立即注册

QQ登录

只需一步，快速开始

1 2 3 4 5
168大数据»社区 › 1F 大数据(BigData) › 流式&日志引擎 › Hadoop生态圈-Kafka的完全分布式部署
返回列表 发新帖
打印 上一主题 下一主题
开启左侧

[Kafka] Hadoop生态圈-Kafka的完全分布式部署

[复制链接]
跳转到指定楼层
楼主
发表于 2018-12-21 10:50:16 | 只看该作者 回帖奖励 |倒序浏览 |阅读模式

马上注册，结交更多数据大咖，获取更多知识干货，轻松玩转大数据

您需要 登录 才可以下载或查看，没有帐号？立即注册

x
本帖最后由 168主编 于 2018-12-21 17:51 编辑

作者：尹正杰来源：http://www.cnblogs.com/yinzhengjie/p/9209319.html
本文仅作社区分享交流用途，如有侵权，请及时联系删除。

本篇博客主要内容就是搭建Kafka完全分布式，它是在kafka本地模式（https://www.cnblogs.com/yinzhengjie/p/9209058.html）的基础之上进一步实现完全分布式搭建过程。
一.试验环境
　　试验环境共计4台服务器
1>.管理服务器（s101）
2>.Kafka节点二（s102，已经部署好了zookeeper服务）
3>.Kafka节点三（s103，已经部署好了zookeeper服务）
4>.Kafka节点四（s104，已经部署好了zookeeper服务）
二.kafka完全分布式部署
1>.将kafka加压后的安装包发送到其他节点（s102,s103,s104）
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s101 data]$ more `which xrsync.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:[url=http://www.cnblogs.com/yinzhengjie]http://www.cnblogs.com/yinzhengjie[/url]
#EMAIL:y1053419035@qq.com

#判断用户是否传参
if [ $# -lt 1 ];then
    echo "请输入参数";
    exit
fi


#获取文件路径
file=$@

#获取子路径
filename=`basename $file`

#获取父路径
dirpath=`dirname $file`

#获取完整路径
cd $dirpath
fullpath=`pwd -P`

#同步文件到DataNode
for (( i=102;i<=105;i++ ))
do
    #使终端变绿色 
    tput setaf 2
    echo =========== s$i %file ===========
    #使终端变回原来的颜色，即白灰色
    tput setaf 7
    #远程执行命令
    rsync -lr $filename `whoami`@s$i:$fullpath
    #判断命令是否执行成功
    if [ $? == 0 ];then
        echo "命令执行成功"
    fi
done
[yinzhengjie@s101 data]$ 

[yinzhengjie@s101 data]$ more `which xrsync.sh`
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka
=========== s102 %file ===========
命令执行成功
=========== s103 %file ===========
命令执行成功
=========== s104 %file ===========
命令执行成功
=========== s105 %file ===========
命令执行成功
[yinzhengjie@s101 data]$

[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka

[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka_2.11-1.1.0/
=========== s102 %file ===========
命令执行成功
=========== s103 %file ===========
命令执行成功
=========== s104 %file ===========
命令执行成功
=========== s105 %file ===========
命令执行成功
[yinzhengjie@s101 data]$ 

[yinzhengjie@s101 data]$ xrsync.sh /soft/kafka_2.11-1.1.0/
2>.分发环境变量
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s101 data]$ su 
Password: 
[root@s101 data]# xrsync.sh /etc/profile
=========== s102 %file ===========
命令执行成功
=========== s103 %file ===========
命令执行成功
=========== s104 %file ===========
命令执行成功
=========== s105 %file ===========
命令执行成功
[root@s101 data]# exit 
exit
[yinzhengjie@s101 data]$
3>.修zk节点的改配置文件
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s102 ~]$ grep broker.id /soft/kafka/config/server.properties
broker.id=102
[yinzhengjie@s102 ~]$ grep listeners /soft/kafka/config/server.properties | grep -v ^#
listeners=PLAINTEXT://s102:9092
[yinzhengjie@s102 ~]$
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s103 ~]$ grep broker.id /soft/kafka/config/server.properties
broker.id=103
[yinzhengjie@s103 ~]$  grep listeners /soft/kafka/config/server.properties | grep -v ^#
listeners=PLAINTEXT://s103:9092
[yinzhengjie@s103 ~]$
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s104 ~]$ grep broker.id /soft/kafka/config/server.properties
broker.id=104
[yinzhengjie@s104 ~]$ 
[yinzhengjie@s104 ~]$ grep listeners /soft/kafka/config/server.properties | grep -v ^#
listeners=PLAINTEXT://s104:9092
[yinzhengjie@s104 ~]$
4>.进入zookeeper客户端并删除zk的kafka节点数据
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s104 ~]$ zkCli.sh 
Connecting to localhost:2181
2018-06-21 01:23:54,935 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-06-21 01:23:54,938 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=s104
2018-06-21 01:23:54,938 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_131
2018-06-21 01:23:54,940 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-06-21 01:23:54,940 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/soft/jdk1.8.0_131/jre
2018-06-21 01:23:54,940 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/soft/zk/bin/../build/classes:/soft/zk/bin/../build/lib/*.jar:/soft/zk/bin/../lib/slf4j-log4j12-1.7.25.jar:/soft/zk/bin/../lib/slf4j-api-1.7.25.jar:/soft/zk/bin/../lib/netty-3.10.6.Final.jar:/soft/zk/bin/../lib/log4j-1.2.17.jar:/soft/zk/bin/../lib/jline-0.9.94.jar:/soft/zk/bin/../lib/audience-annotations-0.5.0.jar:/soft/zk/bin/../zookeeper-3.4.12.jar:/soft/zk/bin/../src/java/lib/*.jar:/soft/zk/bin/../conf:
2018-06-21 01:23:54,940 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-06-21 01:23:54,940 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2018-06-21 01:23:54,940 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2018-06-21 01:23:54,940 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2018-06-21 01:23:54,941 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2018-06-21 01:23:54,941 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-327.el7.x86_64
2018-06-21 01:23:54,941 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=yinzhengjie
2018-06-21 01:23:54,941 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/yinzhengjie
2018-06-21 01:23:54,941 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/yinzhengjie
2018-06-21 01:23:54,942 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@277050dc
Welcome to ZooKeeper!
JLine support is enabled
2018-06-21 01:23:54,973 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1028] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-06-21 01:23:55,031 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-06-21 01:23:55,049 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x6800003ae7350004, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /                                                                                                                                                      
[a, cluster, controller, brokers, zookeeper, yarn-leader-election, hadoop-ha, admin, isr_change_notification, log_dir_event_notification, controller_epoch, consumers, latest_producer_id_block, config, hbase]
[zk: localhost:2181(CONNECTED) 1] rmr /controller /brokers /admin /controller_epoch /consumers /latest_producer_id_block /config /isr_change_notification /cluster /log_dir_event_notification  
[zk: localhost:2181(CONNECTED) 2]
5>.分别启动s102-s104的kafka
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s102 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
[yinzhengjie@s102 ~]$
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s103 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
[yinzhengjie@s103 ~]$
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s104 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
[yinzhengjie@s104 ~]$

6>.创建主题
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s104 ~]$ kafka-topics.sh --zookeeper s102:2181 --list
yinzhengjie
[yinzhengjie@s104 ~]$
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s104 ~]$ kafka-topics.sh --zookeeper s104:2181 --create  --partitions 2 --replication-factor 1  --topic yzj
Created topic "yzj".
[yinzhengjie@s104 ~]$
7>.在任意zk节点开启控制台生产者（例如：在s102上）
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s102 ~]$ kafka-server-start.sh -daemon /soft/kafka/config/server.properties
[yinzhengjie@s102 ~]$ kafka-console-producer.sh --broker-list s104:9092 --topic yzj
>尹正杰到此一游！
>
8>.在任意zk节点开启控制台消费者（例如：在s103上）
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s103 ~]$ kafka-console-consumer.sh --zookeeper s102:2181 --topic yzj --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
尹正杰到此一游！
三.编写kafka启动脚本（“/usr/local/bin/xkafka.sh”，别忘记添加执行权限，而且需要你提前配置好秘钥对哟！）
[AppleScript] 纯文本查看 复制代码
[yinzhengjie@s101 ~]$ more /usr/local/bin/xkafka.sh 
#!/bin/bash
#@author :yinzhengjie
#blog:[url=http://www.cnblogs.com/yinzhengjie]http://www.cnblogs.com/yinzhengjie[/url]
#EMAIL:y1053419035@qq.com

#判断用户是否传参
if [ $# -ne 1 ];then
    echo "无效参数，用法为: $0  {start|stop}"
    exit
fi

#获取用户输入的命令
cmd=$1



for (( i=102 ; i<=104 ; i++ )) ; do
    tput setaf 2
    echo ========== s$i  $cmd ================
    tput setaf 9
    case $cmd in
        start) 
            ssh s$i  "source /etc/profile ; kafka-server-start.sh -daemon /soft/kafka/config/server.properties" 
            echo  s$i  "服务已启动"
            ;;
        stop) 
            ssh s$i  "source /etc/profile ; kafka-server-stop.sh" 
            echo s$i  "服务已停止"
            ;;
            *) 
            echo "无效参数，用法为: $0  {start|stop}"
            exit 
            ;;
     esac
done

[yinzhengjie@s101 ~]$ sudo chmod a+x /usr/local/bin/xkafka.sh 
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ ll /usr/local/bin/xkafka.sh 
-rwxr-xr-x 1 root root 742 Jun 21 02:02 /usr/local/bin/xkafka.sh
[yinzhengjie@s101 ~]$



楼主热帖
分享到:  QQ好友和群QQ好友和群 QQ空间QQ空间 腾讯微博腾讯微博 腾讯朋友腾讯朋友
收藏收藏 转播转播 分享分享 分享淘帖 赞赞 踩踩

168大数据 - 论坛版权1.本主题所有言论和图片纯属网友个人见解，与本站立场无关
2.本站所有主题由网友自行投稿发布。若为首发或独家，该帖子作者与168大数据享有帖子相关版权。
3.其他单位或个人使用、转载或引用本文时必须同时征得该帖子作者和168大数据的同意，并添加本文出处。
4.本站所收集的部分公开资料来源于网络，转载目的在于传递价值及用于交流学习，并不代表本站赞同其观点和对其真实性负责，也不构成任何其他建议。
5.任何通过此网页连接而得到的资讯、产品及服务，本站概不负责，亦不负任何法律责任。
6.本站遵循行业规范，任何转载的稿件都会明确标注作者和来源，若标注有误或遗漏而侵犯到任何版权问题，请尽快告知，本站将及时删除。
7.168大数据管理员和版主有权不事先通知发贴者而删除本文。
回复

使用道具 举报

返回列表 发新帖
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

站长推荐上一条 /1 下一条

关于我们|小黑屋|Archiver|168大数据 ( 京ICP备14035423号 ) |申请友情链接

GMT+8, 2024-5-2 11:01

Powered by BI168大数据社区

© 2012-2014 168大数据

快速回复 返回顶部 返回列表