分布式架构——第10篇:三台主机构建ZooKeeper集群(Docker+MACVLAN)

在拥有两台物理主机的情况下(Docker Host为192.168.1.3/24、Develop PC为192.168.1.2/24),本文使用dockermacvlan技术,对ZooKeeper集群进一步改进,使得外部主机可以直接访问ZooKeeper集群。具体架构如下图:

同时,集群对外表现为三台独立的物理主机,分别拥有各自固定的IP地址MAC地址

启动ZooKeeper容器(主机1 macvlan_zk1)

启动容器时,指定使用macvlan网络(前一篇文章定义的pub_net网络),指定容器IP地址为192.168.1.10。
e.g.

$ sudo docker run --net=pub_net --ip=192.168.1.10 --name macvlan_zk1 --restart always -d zookeeper
4bb7840e0ef1d734ef18dc65149b99533dc214469ad71a773940f869daeeb107

配置zoo.cfg(主机1 macvlan_zk1)

使用docker exec登录到容器内部,配置zookeeper分布式集群。这里直接使用的是docker官方的zookeeper镜像,配置文件zoo.cfg位于/conf/zoo.cfg,具体可参见 https://docs.docker.com/samples/library/zookeeper/

$ sudo docker exec -it macvlan_zk1 /bin/bash
bash-4.4# echo 1 >> /data/myid
bash-4.4# cat /data/myid
1
bash-4.4# vi /conf/zoo.cfg
bash-4.4# exit
exit

以下是一份zoo.cfg的配置内容。
e.g.

clientPort=2181
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60

#add by qingdujun, START, 2018.10.23
server.1=192.168.1.10:2888:3888
server.2=192.168.1.11:2888:3888
server.3=192.168.1.12:2888:3888
#END

配置剩余的两台主机(macvlan_zk2、macvlan_zk3)

以下基本为重复主机1(macvlan_zk1)的配置过程。但是以下几点需要注意:

  • --ip 启动容器时,注意指定不同的IP地址
  • --name 更改一下容器命名。
  • /data/myid 内容应该分别为对应的主机号。
  • zoo.cfg 内容需要保持与主机1 macvlan_zk1完全一致。

下面给出两个示例:

  1. 主机2(macvlan_zk2)
    e.g.

    $ sudo docker run --net=pub_net --ip=192.168.1.11 --name macvlan_zk2 --restart always -d zookeeper
    $ sudo docker exec -it macvlan_zk2 /bin/bash
    bash-4.4# echo 2 >> /data/myid
    bash-4.4# cat /data/myid
    2
    bash-4.4# vi /conf/zoo.cfg
    bash-4.4# exit
    exit
  2. 主机3(macvlan_zk3)
    e.g.

    $ sudo docker run --net=pub_net --ip=192.168.1.12 --name macvlan_zk3 --restart always -d zookeeper
    $ sudo docker exec -it macvlan_zk3 /bin/bash
    bash-4.4# echo 3 >> /data/myid
    bash-4.4# cat /data/myid
    3
    bash-4.4# vi /conf/zoo.cfg
    bash-4.4# exit
    exit

集群状态

由于这里使用的是macvlan网络,Docker HostDocker macvlan(也就是ZooKeeper集群)是无法ping通的。除此之外,其他都可以ping通。

  • Develop PC上可以ping通任何主机。
  • Docker macvlan内部可以相互ping通(亦即,ZooKeeper集群可以相互访问)。
  • Docker Host可以ping通Develop PC
$ echo stat | nc 192.168.1.10 2181
Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
/192.168.1.2:52407[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 3
Sent: 2
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: follower
Node count: 4
$ echo stat | nc 192.168.1.11 2181
Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
/192.168.1.2:52408[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: leader
Node count: 4
Proposal sizes last/min/max: -1/-1/-1
$ echo stat | nc 192.168.1.12 2181
Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
/192.168.1.2:52409[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: follower
Node count: 4

番外

集群搭建时,弄错了两个路径conf/zoo.cfg/conf/zoo.cfg。我一直配置的是错误的路径,导致集群起不来(或者说是配置文件无效),一直显示standalone

flythread@pzz:~$ echo stat | nc 192.168.1.10 2181
Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
/192.168.1.1:58820[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: standalone
Node count: 4

为此,我做了很多折腾,比如将net.ipv4.ip_forward设置为1

$ sudo vi /etc/sysctl.conf
$ sudo sysctl -p /etc/sysctl.conf
net.ipv4.ip_forward = 1

另外,还以为是虚拟网卡物理网卡的原因,在zoo.cfg中设置了quorumListenOnAllIPs=true

最后,补充一下如何停止ZooKeeper集群。首先需要停止restart,然后停止容器,最后删除容器。
e.g.

$ sudo docker update --restart=no macvlan_zk1
$ sudo docker stop macvlan_zk1
$ sudo docker rm macvlan_zk1

当然,也可以在启动容器时设置--restart=unless-stopped这样就可以直接stop容器了。

References:
[1] https://stackoverflow.com/questions/37599128/docker-how-do-you-disable-auto-restart-on-a-container
[2] https://docs.docker.com/samples/library/zookeeper/#configuration
[3] https://blog.csdn.net/alicc/article/details/82773284
[4] http://zookeeper.apache.org/doc/current/zookeeperStarted.html#sc_RunningReplicatedZooKeeper
[5] https://github.com/31z4/zookeeper-docker/tree/c6ef4c1ada3c59dc3c1aaef43619b6e048d3c9e8
[6] http://www.dczou.com/viemall/818.html
[7] https://jiajially.gitbooks.io/dockerguide/chapter_examples/zookeeper_3.html
[8] https://www.cnblogs.com/xhyan/p/6593075.html
[9] https://www.jianshu.com/p/9a2963c2998a
[10] https://cizixs.com/2017/02/14/network-virtualization-macvlan/
[11] https://codeday.me/bug/20180930/267037.html
[12] https://blog.csdn.net/u014686399/article/details/77650353?utm_source=blogxgwz2
[13] https://blog.csdn.net/u014284000/article/details/74508963