歡迎來到Linux教程網
設為首頁 加入收藏
Linux教程網
Linux教程網
Linux教程網
Linux教程網 >> Linux基礎 >> Linux教程 >> docker高級應用之多台主機網絡互聯

docker高級應用之多台主機網絡互聯

日期：2017/2/27 15:55:27   编辑：Linux教程

目前docker主要應用於單機環境，使用網橋模式，但如果想把多台主機網絡互相，讓多台主機內部的container互相通信，就得使用其他的軟件來幫忙，可以使用Weave、Kubernetes、Flannel、SocketPlane或者openvswitch等，我這裡就使用openvswitch來介紹docker多台主機網絡互通。

先看一個使用openvswitch連接的架構圖，連接的方式是vxlan

說明：

這裡有2台主機，分別是NODEA與NODEB，系統是centos7，內核是3.18（默認centos7內核是3.10，但想使用vxlan，所以得升級，參考http://dl528888.blog.51cto.com/2382721/1609850）

docker是1.3.2版本，存儲引擎是devicemapper。

每台主機裡都有2個網橋ovs1與ovs2，ovs1是管理網絡，連接內網網卡em1，ovs2是數據網絡，docker測試機都連接這個ovs2，並且container創建的時候網絡都是none，使用pipework指定固定ip。

然後2台主機使用vxlan連接網絡。

重要：

我個人認為使用這個模式並且指定固定ip，適用於的環境主要是給研發或者個人的測試模式，如果是集群環境，沒必要指定固定ip（我這裡的集群就沒有使用固定ip，使用動態ip，效果很好，後續給大家介紹集群）。

下面是部署方法

環境

一、安裝openvswitch

我的版本是最新的2.3.1

1、安裝基礎環境 
yum install gcc make python-devel openssl-devel kernel-devel graphviz \
   kernel-debug-devel autoconf automake rpm-build redhat-rpm-config \
   libtool
2、下載最新的包 
wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gz
3、解壓與打包 
tar zxvf openvswitch-2.3.1.tar.gz
mkdir -p ~/rpmbuild/SOURCES
cp openvswitch-2.3.1.tar.gz ~/rpmbuild/SOURCES/
sed 's/openvswitch-kmod, //g' openvswitch-2.3.1/rhel/openvswitch.spec > openvswitch-2.3.1/rhel/openvswitch_no_kmod.spec
rpmbuild -bb --without check openvswitch-2.3.1/rhel/openvswitch_no_kmod.spec
之後會在~/rpmbuild/RPMS/x86_64/裡有2個文件 
total 9500
-rw-rw-r-- 1 ovswitch ovswitch 2013688 Jan 15 03:20 openvswitch-2.3.1-1.x86_64.rpm
-rw-rw-r-- 1 ovswitch ovswitch 7712168 Jan 15 03:20 openvswitch-debuginfo-2.3.1-1.x86_64.rpm
安裝第一個就行

4、安裝

yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-2.3.1-1.x86_64.rpm

5、啟動

systemctl start openvswitch

6、查看狀態

[root@docker-test3 tmp]# systemctl status openvswitch
openvswitch.service - LSB: Open vSwitch switch
   Loaded: loaded (/etc/rc.d/init.d/openvswitch)
   Active: active (running) since Wed 2015-01-28 23:34:01 CST; 6 days ago
   CGroup: /system.slice/openvswitch.service
           ├─20314 ovsdb-server: monitoring pid 20315 (healthy)
           ├─20315 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,p...
           ├─20324 ovs-vswitchd: monitoring pid 20325 (healthy)
           └─20325 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log...
 
Jan 28 23:34:01 ip-10-10-17-3 openvswitch[20291]: /etc/openvswitch/conf.db does not exist ... (warning).
Jan 28 23:34:01 ip-10-10-17-3 openvswitch[20291]: Creating empty database /etc/openvswitch/conf.db [  OK  ]
Jan 28 23:34:01 ip-10-10-17-3 openvswitch[20291]: Starting ovsdb-server [  OK  ]
Jan 28 23:34:01 ip-10-10-17-3 ovs-vsctl[20316]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.6.2
Jan 28 23:34:01 ip-10-10-17-3 ovs-vsctl[20321]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.3.1 "external-ids:system-id=\"6ea..."unknown\""
Jan 28 23:34:01 ip-10-10-17-3 openvswitch[20291]: Configuring Open vSwitch system IDs [  OK  ]
Jan 28 23:34:01 ip-10-10-17-3 openvswitch[20291]: Starting ovs-vswitchd [  OK  ]
Jan 28 23:34:01 ip-10-10-17-3 openvswitch[20291]: Enabling remote OVSDB managers [  OK  ]
Jan 28 23:34:01 ip-10-10-17-3 systemd[1]: Started LSB: Open vSwitch switch.
Hint: Some lines were ellipsized, use -l to show in full.

可以看到是正常運行狀態

具體的安裝詳細步驟可以參考

https://github.com/openvswitch/ovs/blob/master/INSTALL.RHEL.md與http://www.linuxidc.com/Linux/2014-12/110272.htm

二、部署單機環境的docker

1、下載pipework

使用這個軟件進行固定ip設置

cd /tmp/
git clone https://github.com/jpetazzo/pipework.git

2、在NODEA（ip是10.10.17.3）裡運行下面命令

可以把下面內容復制到腳本裡運行

#!/bin/bash
#author: Deng Lei
#email: [email protected]
#刪除docker測試機
docker rm `docker stop $(docker ps -a -q)`
#刪除已有的openvswitch交換機
ovs-vsctl list-br|xargs -I {} ovs-vsctl del-br {}
#創建交換機
ovs-vsctl add-br ovs1
ovs-vsctl add-br ovs2
#把物理網卡加入ovs2
ovs-vsctl add-port ovs1 em1
ip link set ovs1 up
ifconfig em1 0
ifconfig ovs1 10.10.17.3
ip link set ovs2 up
ip addr add 172.16.0.3/16 dev ovs2
 
pipework_dir='/tmp/pipework'
docker run --restart always --privileged -d  --net="none" --name='test1' docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord
$pipework_dir/pipework ovs2 test1 172.16.0.5/[email protected]
 
docker run --restart always --privileged -d  --net="none" --name='test2' docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord
$pipework_dir/pipework ovs2 test2 172.16.0.6/[email protected]

根據自己的環境修改上面內容

運行腳本

[root@docker-test3 tmp]# sh openvswitch_docker.sh
5a1139276ccd
03d866e20f58
6352f9ecd69450e332a13ec5dfecba106e58cf2a301e9b539a7e690cd61934f7
8e294816e5225dbee7e8442be1d9d5b4c6072d935b68e332b84196ea1db6f07c
[root@docker-test3 tmp]# docker ps -a
CONTAINER ID        IMAGE                                          COMMAND                CREATED             STATUS              PORTS               NAMES
8e294816e522        docker.ops-chukong.com:5000/centos6-http:new   "/usr/bin/supervisor   3 seconds ago       Up 2 seconds                            test2
6352f9ecd694        docker.ops-chukong.com:5000/centos6-http:new   "/usr/bin/supervisor   3 seconds ago       Up 3 seconds

可以看到已經啟動了2個容器，分別是test1與test2

下面從本地登陸指定的ip試試

[root@docker-test3 tmp]# ssh 172.16.0.5
The authenticity of host '172.16.0.5 (172.16.0.5)' can't be established.
RSA key fingerprint is 39:7c:13:9f:d4:b0:d7:63:fc:ff:ae:e3:46:a4:bf:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.0.5' (RSA) to the list of known hosts.
[email protected]'s password:
Last login: Mon Nov 17 14:10:39 2014 from 172.17.42.1
root@6352f9ecd694:~
18:57:50 # ifconfig
eth1      Link encap:Ethernet  HWaddr 26:39:B1:88:25:CC
          inet addr:172.16.0.5  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::2439:b1ff:fe88:25cc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:65 errors:0 dropped:6 overruns:0 frame:0
          TX packets:41 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8708 (8.5 KiB)  TX bytes:5992 (5.8 KiB)
 
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
 
root@6352f9ecd694:~
18:57:51 # ping 172.16.0.6 -c 2
PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data.
64 bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=0.433 ms
64 bytes from 172.16.0.6: icmp_seq=2 ttl=64 time=0.040 ms
 
--- 172.16.0.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.040/0.236/0.433/0.197 ms
root@6352f9ecd694:~
18:58:03 # ping 172.16.0.3 -c 2
PING 172.16.0.3 (172.16.0.3) 56(84) bytes of data.
64 bytes from 172.16.0.3: icmp_seq=1 ttl=64 time=0.369 ms
64 bytes from 172.16.0.3: icmp_seq=2 ttl=64 time=0.045 ms
 
--- 172.16.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.045/0.207/0.369/0.162 ms
root@6352f9ecd694:~
18:58:09 # ping www.baidu.com -c 2
PING www.a.shifen.com (180.149.131.205) 56(84) bytes of data.
64 bytes from 180.149.131.205: icmp_seq=1 ttl=54 time=1.83 ms
64 bytes from 180.149.131.205: icmp_seq=2 ttl=54 time=1.81 ms
 
--- www.a.shifen.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 1.816/1.827/1.839/0.044 ms
登陸後可以看到容器內的ip是指定的，並且能ping另外同一個網段的172.16.0.6，外網也能ping通。

下面進行vxlan測試，需要現在另外一個物理宿主機進行上面的腳本安裝，然後在進行vxlan配置

3、在NODEB（ip是10.10.17.4）裡運行

腳本內容是 
#!/bin/bash
#author: Deng Lei
#email: [email protected]
#刪除docker測試機
docker rm `docker stop $(docker ps -a -q)`
#刪除已有的openvswitch交換機
ovs-vsctl list-br|xargs -I {} ovs-vsctl del-br {}
#創建交換機
ovs-vsctl add-br ovs1
ovs-vsctl add-br ovs2
#把物理網卡加入ovs2
ovs-vsctl add-port ovs1 em1
ip link set ovs1 up
ifconfig em1 0
ifconfig ovs1 10.10.17.4
ip link set ovs2 up
ip addr add 172.16.0.4/16 dev ovs2
 
pipework_dir='/tmp/pipework'
docker run --restart always --privileged -d  --net="none" --name='test1' docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord
$pipework_dir/pipework ovs2 test1 172.16.0.8/[email protected]
 
docker run --restart always --privileged -d  --net="none" --name='test2' docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord
$pipework_dir/pipework ovs2 test2 172.16.0.9/[email protected]
運行這個腳本 
[root@docker-test4 tmp]# sh openvswitch_docker.sh
3999d60c5833
1b42d09f3311
a10c7b6f1141056e5276c44b348652e66921f322a96699903cf1c372858633d3
7e907cf62e593c27deb3ff455d1107074dfb975e68a3ac42ff844e84794322cd
[root@docker-test4 tmp]# docker ps -a
CONTAINER ID        IMAGE                                          COMMAND                CREATED             STATUS              PORTS               NAMES
7e907cf62e59        docker.ops-chukong.com:5000/centos6-http:new   "/usr/bin/supervisor   3 seconds ago       Up 2 seconds                            test2
a10c7b6f1141        docker.ops-chukong.com:5000/centos6-http:new   "/usr/bin/supervisor   4 seconds ago       Up 3 seconds
登陸分別的固定ip試試 
[root@docker-test4 tmp]# ssh 172.16.0.8
The authenticity of host '172.16.0.8 (172.16.0.8)' can't be established.
RSA key fingerprint is 39:7c:13:9f:d4:b0:d7:63:fc:ff:ae:e3:46:a4:bf:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.0.8' (RSA) to the list of known hosts.
[email protected]'s password:
Last login: Mon Nov 17 14:10:39 2014 from 172.17.42.1
root@a10c7b6f1141:~
18:45:05 # ifconfig
eth1      Link encap:Ethernet  HWaddr CA:46:87:58:6C:BF
          inet addr:172.16.0.8  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::c846:87ff:fe58:6cbf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:75 errors:0 dropped:2 overruns:0 frame:0
          TX packets:41 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10787 (10.5 KiB)  TX bytes:5992 (5.8 KiB)
 
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
 
root@a10c7b6f1141:~
18:45:06 # ping 172.16.0.9
PING 172.16.0.9 (172.16.0.9) 56(84) bytes of data.
64 bytes from 172.16.0.9: icmp_seq=1 ttl=64 time=0.615 ms
^C
--- 172.16.0.9 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 531ms
rtt min/avg/max/mdev = 0.615/0.615/0.615/0.000 ms
root@a10c7b6f1141:~
18:45:10 # ping 172.16.0.4
PING 172.16.0.4 (172.16.0.4) 56(84) bytes of data.
64 bytes from 172.16.0.4: icmp_seq=1 ttl=64 time=0.270 ms
^C
--- 172.16.0.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 581ms
rtt min/avg/max/mdev = 0.270/0.270/0.270/0.000 ms
root@a10c7b6f1141:~
18:45:12 # ping www.baidu.com -c 2
PING www.a.shifen.com (180.149.131.236) 56(84) bytes of data.
64 bytes from 180.149.131.236: icmp_seq=1 ttl=54 time=1.90 ms
64 bytes from 180.149.131.236: icmp_seq=2 ttl=54 time=2.00 ms
 
--- www.a.shifen.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 1.900/1.950/2.000/0.050 ms
可以看到結果跟NODEA（10.10.17.3）裡運行的一樣，登陸後可以看到容器內的ip是指定的，並且能ping另外同一個網段的172.16.0.9，外網也能ping通

然後在試試能否ping通對方的em1網卡與對方ovs2的ip

4、在NODEA裡測試 
root@6352f9ecd694:~
18:58:48 # ping 10.10.17.3 -c 2
PING 10.10.17.3 (10.10.17.3) 56(84) bytes of data.
64 bytes from 10.10.17.3: icmp_seq=1 ttl=64 time=0.317 ms
64 bytes from 10.10.17.3: icmp_seq=2 ttl=64 time=0.042 ms
 
--- 10.10.17.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.042/0.179/0.317/0.138 ms
root@6352f9ecd694:~
18:58:52 # ping 10.10.17.4 -c 2
PING 10.10.17.4 (10.10.17.4) 56(84) bytes of data.
64 bytes from 10.10.17.4: icmp_seq=1 ttl=63 time=1.35 ms
64 bytes from 10.10.17.4: icmp_seq=2 ttl=63 time=0.271 ms
 
--- 10.10.17.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.271/0.814/1.357/0.543 ms
root@6352f9ecd694:~
18:58:56 # ping 172.16.0.3 -c 2
PING 172.16.0.3 (172.16.0.3) 56(84) bytes of data.
64 bytes from 172.16.0.3: icmp_seq=1 ttl=64 time=0.330 ms
64 bytes from 172.16.0.3: icmp_seq=2 ttl=64 time=0.040 ms
 
--- 172.16.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.040/0.185/0.330/0.145 ms
root@6352f9ecd694:~
18:59:04 # ping 172.16.0.4 -c 2
PING 172.16.0.4 (172.16.0.4) 56(84) bytes of data.
From 172.16.0.5 icmp_seq=1 Destination Host Unreachable
From 172.16.0.5 icmp_seq=2 Destination Host Unreachable
 
--- 172.16.0.4 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 3007ms
pipe 2
能ping通自己的em1與10.10.17.4的em1網卡，並且對方的ovs2的ip也能ping通，但ovs2裡的主機無法ping通

5、在NODEB裡測試

root@a10c7b6f1141:~
18:59:35 # ping 10.10.17.4 -c2
PING 10.10.17.4 (10.10.17.4) 56(84) bytes of data.
64 bytes from 10.10.17.4: icmp_seq=1 ttl=64 time=0.306 ms
64 bytes from 10.10.17.4: icmp_seq=2 ttl=64 time=0.032 ms
 
--- 10.10.17.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.032/0.169/0.306/0.137 ms
root@a10c7b6f1141:~
18:59:48 # ping 10.10.17.3 -c2
PING 10.10.17.3 (10.10.17.3) 56(84) bytes of data.
64 bytes from 10.10.17.3: icmp_seq=1 ttl=63 time=0.752 ms
64 bytes from 10.10.17.3: icmp_seq=2 ttl=63 time=0.268 ms
 
--- 10.10.17.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.268/0.510/0.752/0.242 ms
root@a10c7b6f1141:~
18:59:51 # ping 172.16.0.4 -c2
PING 172.16.0.4 (172.16.0.4) 56(84) bytes of data.
64 bytes from 172.16.0.4: icmp_seq=1 ttl=64 time=0.215 ms
64 bytes from 172.16.0.4: icmp_seq=2 ttl=64 time=0.037 ms
 
--- 172.16.0.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.037/0.126/0.215/0.089 ms
root@a10c7b6f1141:~
18:59:57 # ping 172.16.0.3 -c2
PING 172.16.0.3 (172.16.0.3) 56(84) bytes of data.
From 172.16.0.8 icmp_seq=1 Destination Host Unreachable
From 172.16.0.8 icmp_seq=2 Destination Host Unreachable
 
--- 172.16.0.3 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 3006ms
pipe 2
結果也是一樣，能ping通自己的em1與NODEA（10.10.17.3）的em1網卡，並且對方的ovs2的ip也能ping通，但ovs2裡的主機無法ping通

6、vxlan設置

在NODEA裡運行

ovs-vsctl add-port ovs2 vx1 -- set interface vx1 type=vxlan options:remote_ip=10.10.17.4
[root@docker-test3 tmp]# ovs-vsctl show
d895d78b-8c89-49bc-b429-da6a4a2dcb3a
    Bridge "ovs1"
        Port "em1"
            Interface "em1"
        Port "ovs1"
            Interface "ovs1"
                type: internal
    Bridge "ovs2"
        Port "veth1pl15561"
            Interface "veth1pl15561"
        Port "veth1pl15662"
            Interface "veth1pl15662"
        Port "vx1"
            Interface "vx1"
                type: vxlan
                options: {remote_ip="10.10.17.4"}
        Port "ovs2"
            Interface "ovs2"
                type: internal
    ovs_version: "2.3.1"

在NODEB裡運行

ovs-vsctl add-port ovs2 vx1 -- set interface vx1 type=vxlan options:remote_ip=10.10.17.3
[root@docker-test4 tmp]# ovs-vsctl show
5a4b1bcd-3a91-4670-9c60-8fabbff37e85
    Bridge "ovs2"
        Port "veth1pl28665"
            Interface "veth1pl28665"
        Port "ovs2"
            Interface "ovs2"
                type: internal
        Port "vx1"
            Interface "vx1"
                type: vxlan
                options: {remote_ip="10.10.17.3"}
        Port "veth1pl28766"
            Interface "veth1pl28766"
    Bridge "ovs1"
        Port "ovs1"
            Interface "ovs1"
                type: internal
        Port "em1"
            Interface "em1"
    ovs_version: "2.3.1"

現在NODEA與NODEB這2台物理機的網絡都是互通的，容器的網絡也是互通。

然後在NODEA（10.10.17.3）裡ping NODEB（10.10.17.4）的ovs2 ip與容器的ip 
[root@docker-test3 tmp]# ssh 172.16.0.5
[email protected]'s password:
Last login: Tue Feb  3 19:04:30 2015 from 172.16.0.3
root@6352f9ecd694:~
19:04:38 # ping 172.16.0.4 -c 2
PING 172.16.0.4 (172.16.0.4) 56(84) bytes of data.
64 bytes from 172.16.0.4: icmp_seq=1 ttl=64 time=0.623 ms
64 bytes from 172.16.0.4: icmp_seq=2 ttl=64 time=0.272 ms
 
--- 172.16.0.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.272/0.447/0.623/0.176 ms
root@6352f9ecd694:~
19:04:41 # ping 172.16.0.8 -c 2
PING 172.16.0.8 (172.16.0.8) 56(84) bytes of data.
64 bytes from 172.16.0.8: icmp_seq=1 ttl=64 time=1.76 ms
64 bytes from 172.16.0.8: icmp_seq=2 ttl=64 time=0.277 ms
 
--- 172.16.0.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.277/1.023/1.769/0.746 ms
root@6352f9ecd694:~
19:04:44 # ping 172.16.0.9 -c 2
PING 172.16.0.9 (172.16.0.9) 56(84) bytes of data.
64 bytes from 172.16.0.9: icmp_seq=1 ttl=64 time=1.83 ms
64 bytes from 172.16.0.9: icmp_seq=2 ttl=64 time=0.276 ms
 
--- 172.16.0.9 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.276/1.053/1.830/0.777 ms
可以看到可以在NODEA（10.10.17.3）裡ping通NODEB（10.10.17.4）的ovs2 ip與交換機下面的容器ip如果各自設置vxlan，還是無法連接請看看iptables裡是否給ovs1進行了input放行 
[root@docker-test3 tmp]# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Fri Dec  6 10:59:13 2013
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1:83]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i em1 -j ACCEPT
-A INPUT -i ovs1 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 50020 -j ACCEPT
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A FORWARD -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -m limit --limit 1/sec -j ACCEPT
COMMIT
# Completed on Fri Dec  6 10:59:13 2013
*nat
:PREROUTING ACCEPT [2:269]
:POSTROUTING ACCEPT [1739:127286]
:OUTPUT ACCEPT [1739:127286]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.16.0.0/8 ! -d 172.16.0.0/8 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
COMMIT
在NODEB裡測試 
[root@docker-test4 tmp]# ssh 172.16.0.8
[email protected]'s password:
Last login: Tue Feb  3 18:59:35 2015 from 172.16.0.4
root@a10c7b6f1141:~
19:08:08 # ping 172.16.0.3 -c 2
PING 172.16.0.3 (172.16.0.3) 56(84) bytes of data.
64 bytes from 172.16.0.3: icmp_seq=1 ttl=64 time=1.48 ms
64 bytes from 172.16.0.3: icmp_seq=2 ttl=64 time=0.289 ms
 
--- 172.16.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.289/0.889/1.489/0.600 ms
root@a10c7b6f1141:~
19:08:13 # ping 172.16.0.5 -c 2
PING 172.16.0.5 (172.16.0.5) 56(84) bytes of data.
64 bytes from 172.16.0.5: icmp_seq=1 ttl=64 time=1.27 ms
64 bytes from 172.16.0.5: icmp_seq=2 ttl=64 time=0.289 ms
 
--- 172.16.0.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.289/0.783/1.277/0.494 ms
root@a10c7b6f1141:~
19:08:16 # ping 172.16.0.6 -c 2
PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data.
64 bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=1.32 ms
64 bytes from 172.16.0.6: icmp_seq=2 ttl=64 time=0.275 ms
 
--- 172.16.0.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.275/0.800/1.326/0.526 ms
結果也是一樣，設置了vxlan就可以2個宿主機的所有服務器進行通信。

目前是2個節點的vxlan，如果是3個節點呢

7、vxlan多節點應用（超過2個節點）

架構圖為

wKioL1TRytzS5IA5AAEDkQTjbok653.jpg

新節點是NODEC（ip是10.10.21.199）

環境為

wKiom1TRyw6ivPMVAAEKGTIehXU599.jpg

部署單機環境，腳本內容是

#!/bin/bash
#author: Deng Lei
#email: [email protected]
#刪除docker測試機
docker rm `docker stop $(docker ps -a -q)`
#刪除已有的openvswitch交換機
ovs-vsctl list-br|xargs -I {} ovs-vsctl del-br {}
#創建交換機
ovs-vsctl add-br ovs1
ovs-vsctl add-br ovs2
#把物理網卡加入ovs2
ovs-vsctl add-port ovs1 em1
ip link set ovs1 up
ifconfig em1 0
ifconfig ovs1 10.10.21.199
ip link set ovs2 up
ip addr add 172.16.0.11/16 dev ovs2
 
pipework_dir='/tmp/pipework'
docker run --restart always --privileged -d  --net="none" --name='test1' docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord
$pipework_dir/pipework ovs2 test1 172.16.0.12/[email protected]
 
docker run --restart always --privileged -d  --net="none" --name='test2' docker.ops-chukong.com:5000/centos6-http:new /usr/bin/supervisord
$pipework_dir/pipework ovs2 test2 172.16.0.13/[email protected]
運行腳本 
19:14:13 # sh openvswitch_docker.sh
534db79be57a
c014d47212868080a2a3c473b916ccbc9a45ecae4ab564807104faeb7429e1d3
5a98c459438d6e2c37d040f18282509763acc16d63010adc72852585dc255666
測試
root@docker-test1:/tmp
19:17:00 # ssh 172.16.0.12
[email protected]'s password:
Last login: Mon Nov 17 14:10:39 2014 from 172.17.42.1
root@c014d4721286:~
19:17:08 # ifconfig
eth1      Link encap:Ethernet  HWaddr 4A:B3:FE:B7:01:C7
          inet addr:172.16.0.12  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::48b3:feff:feb7:1c7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:117 errors:0 dropped:0 overruns:0 frame:0
          TX packets:89 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:20153 (19.6 KiB)  TX bytes:16600 (16.2 KiB)
 
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
 
root@c014d4721286:~
19:17:10 # ping 172.16.0.11 -c 2
PING 172.16.0.11 (172.16.0.11) 56(84) bytes of data.
64 bytes from 172.16.0.11: icmp_seq=1 ttl=64 time=0.240 ms
64 bytes from 172.16.0.11: icmp_seq=2 ttl=64 time=0.039 ms
 
--- 172.16.0.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.039/0.139/0.240/0.101 ms
root@c014d4721286:~
19:17:16 # ping 172.16.0.12 -c 2
PING 172.16.0.12 (172.16.0.12) 56(84) bytes of data.
64 bytes from 172.16.0.12: icmp_seq=1 ttl=64 time=0.041 ms
64 bytes from 172.16.0.12: icmp_seq=2 ttl=64 time=0.026 ms
 
--- 172.16.0.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.026/0.033/0.041/0.009 ms
root@c014d4721286:~
19:17:18 # ping 172.16.0.13 -c 2
PING 172.16.0.13 (172.16.0.13) 56(84) bytes of data.
64 bytes from 172.16.0.13: icmp_seq=1 ttl=64 time=0.316 ms
64 bytes from 172.16.0.13: icmp_seq=2 ttl=64 time=0.040 ms
 
--- 172.16.0.13 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.040/0.178/0.316/0.138 ms
root@c014d4721286:~
19:17:21 # ping www.baidu.com -c 2
PING www.a.shifen.com (180.149.131.205) 56(84) bytes of data.
64 bytes from 180.149.131.205: icmp_seq=1 ttl=54 time=2.65 ms
64 bytes from 180.149.131.205: icmp_seq=2 ttl=54 time=1.93 ms
 
--- www.a.shifen.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.939/2.295/2.651/0.356 ms
可以看到可以ping通本地的ovs2 的ip與交換機下面是ip 
root@c014d4721286:~
19:17:26 # ping 10.10.17.3 -c 2
PING 10.10.17.3 (10.10.17.3) 56(84) bytes of data.
64 bytes from 10.10.17.3: icmp_seq=1 ttl=63 time=0.418 ms
64 bytes from 10.10.17.3: icmp_seq=2 ttl=63 time=0.213 ms
 
--- 10.10.17.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.213/0.315/0.418/0.104 ms
root@c014d4721286:~
19:18:05 # ping 10.10.17.4 -c 2
PING 10.10.17.4 (10.10.17.4) 56(84) bytes of data.
64 bytes from 10.10.17.4: icmp_seq=1 ttl=63 time=0.865 ms
64 bytes from 10.10.17.4: icmp_seq=2 ttl=63 time=0.223 ms
 
--- 10.10.17.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.223/0.544/0.865/0.321 ms
root@c014d4721286:~
19:18:08 # ping 172.16.0.3 -c 2
PING 172.16.0.3 (172.16.0.3) 56(84) bytes of data.
From 172.16.0.12 icmp_seq=1 Destination Host Unreachable
From 172.16.0.12 icmp_seq=2 Destination Host Unreachable
 
--- 172.16.0.3 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 3006ms
pipe 2
root@c014d4721286:~
19:18:19 # ping 172.16.0.4 -c 2
PING 172.16.0.4 (172.16.0.4) 56(84) bytes of data.
From 172.16.0.12 icmp_seq=1 Destination Host Unreachable
From 172.16.0.12 icmp_seq=2 Destination Host Unreachable
 
--- 172.16.0.4 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 3004ms
pipe 2
可以看到能ping通NODEA（10.10.17.3）與NODEB（10.10.17.4）（em1網卡都是走物理交換機），但他們2個的ovs2都無法ping通下面是在NODEC（10.10.21.199）裡與10.10.17.3做一個vxlan 
19:19:22 # ovs-vsctl add-port ovs2 vx1 -- set interface vx1 type=vxlan options:remote_ip=10.10.17.3
root@docker-test1:/tmp
19:20:01 # ovs-vsctl show
96259d0f-b794-49fd-81fb-3251ede9c2a5
    Bridge "ovs2"
        Port "veth1pl30126"
            Interface "veth1pl30126"
        Port "vx1"
            Interface "vx1"
                type: vxlan
                options: {remote_ip="10.10.17.3"}
        Port "veth1pl30247"
            Interface "veth1pl30247"
        Port "ovs2"
            Interface "ovs2"
                type: internal
    Bridge "ovs1"
        Port "ovs1"
            Interface "ovs1"
                type: internal
        Port "em1"
            Interface "em1"
    ovs_version: "2.3.1"
然後還需要在NODEA（10.10.17.3）裡配置
[root@docker-test3 x86_64]# ovs-vsctl add-port ovs2 vx2 -- set interface vx2 type=vxlan options:remote_ip=10.10.21.199
[root@docker-test3 x86_64]# ovs-vsctl show
d895d78b-8c89-49bc-b429-da6a4a2dcb3a
    Bridge "ovs1"
        Port "em1"
            Interface "em1"
        Port "ovs1"
            Interface "ovs1"
                type: internal
    Bridge "ovs2"
        Port "veth1pl15561"
            Interface "veth1pl15561"
        Port "veth1pl15662"
            Interface "veth1pl15662"
        Port "vx2"
            Interface "vx2"
                type: vxlan
                options: {remote_ip="10.10.21.199"}
        Port "vx1"
            Interface "vx1"
                type: vxlan
                options: {remote_ip="10.10.17.4"}
        Port "ovs2"
            Interface "ovs2"
                type: internal
    ovs_version: "2.3.1"
之前在NODEA（10.10.17.3）裡與NODE（10.10.17.4）做的vxlan使用vx1，這裡NODEA（10.10.17.3）與NODEC（10.10.21.199）就使用vx2端口然後在NODEA（10.10.17.3）裡ping NODEC（10.10.21.199）的ovs2 ip與交換機下面的ip 
[root@docker-test3 x86_64]# ping 172.16.0.11 -c 2
PING 172.16.0.11 (172.16.0.11) 56(84) bytes of data.
64 bytes from 172.16.0.11: icmp_seq=1 ttl=64 time=1.48 ms
64 bytes from 172.16.0.11: icmp_seq=2 ttl=64 time=0.244 ms
 
--- 172.16.0.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.244/0.865/1.487/0.622 ms
[root@docker-test3 x86_64]# ping 172.16.0.12 -c 2
PING 172.16.0.12 (172.16.0.12) 56(84) bytes of data.
64 bytes from 172.16.0.12: icmp_seq=1 ttl=64 time=1.54 ms
64 bytes from 172.16.0.12: icmp_seq=2 ttl=64 time=0.464 ms
 
--- 172.16.0.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.464/1.006/1.549/0.543 ms
[root@docker-test3 x86_64]# ping 172.16.0.13 -c 2
PING 172.16.0.13 (172.16.0.13) 56(84) bytes of data.
64 bytes from 172.16.0.13: icmp_seq=1 ttl=64 time=1.73 ms
64 bytes from 172.16.0.13: icmp_seq=2 ttl=64 time=0.232 ms
 
--- 172.16.0.13 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.232/0.983/1.735/0.752 ms
可以看到是通的在NODEC（10.10.21.199）裡ping NODEA（10.10.17.3）的ovs2的ip與交換機下面的ip 
root@docker-test1:/tmp
19:23:17 # ping 172.16.0.3 -c 2
PING 172.16.0.3 (172.16.0.3) 56(84) bytes of data.
64 bytes from 172.16.0.3: icmp_seq=1 ttl=64 time=0.598 ms
64 bytes from 172.16.0.3: icmp_seq=2 ttl=64 time=0.300 ms
 
--- 172.16.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.300/0.449/0.598/0.149 ms
root@docker-test1:/tmp
19:23:26 # ping 172.16.0.5 -c 2
PING 172.16.0.5 (172.16.0.5) 56(84) bytes of data.
64 bytes from 172.16.0.5: icmp_seq=1 ttl=64 time=1.23 ms
64 bytes from 172.16.0.5: icmp_seq=2 ttl=64 time=0.214 ms
 
--- 172.16.0.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.214/0.726/1.239/0.513 ms
root@docker-test1:/tmp
19:23:29 # ping 172.16.0.6 -c 2
PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data.
64 bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=1.00 ms
64 bytes from 172.16.0.6: icmp_seq=2 ttl=64 time=0.226 ms
 
--- 172.16.0.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.226/0.613/1.000/0.387 ms
root@docker-test1:/tmp
也是通的，然後從NODEC（10.10.21.199） ping NODEB（10.10.17.4）的ovs2的ip與其交換機的ip 
root@docker-test1:/tmp
19:55:05 # ping -c 10 172.16.0.4
PING 172.16.0.4 (172.16.0.4) 56(84) bytes of data.
64 bytes from 172.16.0.4: icmp_seq=1 ttl=64 time=1.99 ms
64 bytes from 172.16.0.4: icmp_seq=2 ttl=64 time=0.486 ms
64 bytes from 172.16.0.4: icmp_seq=3 ttl=64 time=0.395 ms
64 bytes from 172.16.0.4: icmp_seq=4 ttl=64 time=0.452 ms
64 bytes from 172.16.0.4: icmp_seq=5 ttl=64 time=0.457 ms
64 bytes from 172.16.0.4: icmp_seq=6 ttl=64 time=0.461 ms
64 bytes from 172.16.0.4: icmp_seq=7 ttl=64 time=0.457 ms
64 bytes from 172.16.0.4: icmp_seq=8 ttl=64 time=0.428 ms
64 bytes from 172.16.0.4: icmp_seq=9 ttl=64 time=0.492 ms
64 bytes from 172.16.0.4: icmp_seq=10 ttl=64 time=0.461 ms
 
--- 172.16.0.4 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9000ms
rtt min/avg/max/mdev = 0.395/0.608/1.995/0.463 ms
可以看到是通的，平均延遲0.608，並且可以發現使用了vxlan，3個節點，如果想全部互通，只需要2個線連接就行。

如果使用gre模式，3個節點就需要3個線了，架構圖為

wKioL1TRzRyzN9XzAAEiAFU8w0A716.jpg

目前使用docker結合openvswitch的vxlan模式就把多台主機的docker連接起來，這樣很多測試就方便很多，但還是建議把這樣的方式作為測試環境。
原文：http://dl528888.blog.51cto.com/2382721/1611491

Linux教程

利用saltstack部署lnmp環境（yum版本）

一、簡介 saltstack 是一個新基礎設施管理工具，可以看做

Linux 系統監控負載top命令詳解

Ubuntu使用教程：解決VirtualBox中截屏時的黑屏故障

Ubuntu 13.04下載 編譯Android 4.0

安裝Windows7+Ubuntu13.04+CentOS7三系統

Linux系統中安裝mplayer的步驟

Linux中掛載NTFS格式的硬盤

Linux平台Ubuntu系統觸摸板簡單設置上手

Ubuntu安裝JDK及環境變量配置

Linux教程網

JetBrains 加入 .NET 基金會

JetBrains 加入 .NET 基金會JetBrains 加入 .NET 基金會在 BUILD 大會上，微

SUN OS 作業系統簡介

Ubuntu12.04下雙顯卡的配置（N卡）

linux中設置環境變量和gcc/g++的include路徑

用Linux Command Shell模擬DOS命令

FreeBSD下通過PoPToP實現遠程訪問VPN

Linux如何查看so中函數名

Nginx中搭建Nagios監控平台

微軟出陰招陷害 拍賣Linux相關版權

相关文章
Linux基礎知識Linux入門Linux技術Linux教程Linux服務器關於Linux
文章推荐
CentOS 5/6 生產環境若干優化實戰

使用Mondo rescue備份還原linux系統

Linux系統分區：基本原理

如果您使用的是普通分區設
facebook海量圖片存儲系統與淘寶TFS系統比較

热点聚焦

Ubuntu 8.04 LTS 正式發布日期近了

讓Ubuntu更新內核時自動編譯官方版Nvidia驅動模塊

Xubuntu 9.04 Jaunty Jackalope Alpha 3桌面截圖

硬盤或光盤安裝Fedora時選擇EXT4 方法

SUSE Linux登錄時黑屏解決辦法

在 Linux 中怎樣使用cp命令合並目錄樹

ubuntu 7.04 官方更新源

系統引導管理器GRUB，為初學者指南

從boot.ini引導進入ubuntu7.04的解決

Linux 安裝與配置JDK1.7.0_55

Copyright © Linux教程網 All Rights Reserved