多机房OPENVPN互联的解决方法

  • A+
所属分类:Project

一、基础环境

$ uname -a
Linux openvpn-server.t4x.org 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ uname -r
3.10.0-693.el7.x86_64
$ uname -m
x86_64

二、IP信息

杭州机房 : (Centos 7.4)
VPN-SERVER:10.4.0.4(公网IP)、192.168.101.1(内网IP)
机房主机IP:192.168.101.2(无公网IP)

$ ping 192.168.102.1 #ping 广东机房vpn client
PING 192.168.102.1 (192.168.102.1) 56(84) bytes of data.
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ ping 192.168.102.2 #ping 广东机房内网主机
PING 192.168.102.2 (192.168.102.2) 56(84) bytes of data.
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

$ ping 192.168.103.1 #ping 北京机房vpn client
PING 192.168.103.1 (192.168.103.1) 56(84) bytes of data.
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

北京机房 : (Centos 6.8)
VPN-SERVER:10.4.0.8(公网IP)、192.168.103.1(内网IP)
机房主机IP:192.168.103.2(无公网IP)

$ ping 192.168.101.1 #ping 杭州机房vpn server
PING 192.168.101.1 (192.168.101.1) 56(84) bytes of data.
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ ping 192.168.101.2 #ping 杭州机房内网主机
PING 192.168.101.2 (192.168.101.2) 56(84) bytes of data.
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ ping 192.168.102.1 #ping 广东机房vpn client
PING 192.168.102.1 (192.168.102.1) 56(84) bytes of data.
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ ping 192.168.102.2 #ping 广东机房内网主机
PING 192.168.102.2 (192.168.102.2) 56(84) bytes of data.

广东机房:(Centos 7.4)
VPN-SERVER:10.4.0.6(公网IP)、192.168.102.1(内网IP)
机房主机IP:192.168.102.2(无公网IP)

$ ping 192.168.101.1 #ping 杭州机房vpn server
PING 192.168.101.1 (192.168.101.1) 56(84) bytes of data.
3 packets transmitted, 0 received, 100% packet loss, time 2000ms

$ ping 192.168.101.2 #ping 杭州机房内网
PING 192.168.101.2 (192.168.101.2) 56(84) bytes of data.
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ ping 192.168.103.1 #ping 北京机房vpn client
PING 192.168.103.1 (192.168.103.1) 56(84) bytes of data.
1 packets transmitted, 0 received, 100% packet loss, time 0ms


1:由于用vm模拟,需要首先将宿主机上面的网关删除,以免影响实验效果!!!
2:调试前,将firewalld关闭,以免影响实验效果!!!

C:\WINDOWS\system32>route delete 192.168.101.0
操作完成!

C:\WINDOWS\system32>route delete 192.168.102.1
操作完成!

C:\WINDOWS\system32>route delete 192.168.103.0
操作完成!

C:\WINDOWS\system32>route delete 192.168.101.1
操作完成!

C:\WINDOWS\system32>route delete 192.168.102.0
操作完成!

三、OPENVPN服务端安装

四、OPENVPN服务端证书生成

== 此次操作生成的文件/usr/local/openvpn/key/easyrsa3/pki/ca.crt ==

$ ./easyrsa gen-req server nopass #创建服务器端证书,需要输入Common Name ./easyrsa gen-req server此时当启动服务端时候需要输入服务端密码,此次取消passwd

== 此次操作生成的文件 req: /usr/local/openvpn/key/easyrsa3/pki/reqs/server.req ==
== 此次操作生成的文件 key: /usr/local/openvpn/key/easyrsa3/pki/private/server.key ==

$ ./easyrsa sign server server #签约服务端证书,需输入创建ca时的ca密码

==此次操作生成的文件/usr/local/openvpn/key/easyrsa3/pki/issued/server.crt==

$ ./easyrsa gen-dh #创建Diffie-Hellman parameters

五、OPENVPN客户端证书生成

方式1:ca证书在pki目录下

==此次操作生成的文件 req: /usr/local/openvpn/key/easyrsa3/pki/reqs/bj.req ==
==此次操作生成的文件 key: /usr/local/openvpn/key/easyrsa3/pki/private/bj.key ==

$ ./easyrsa gen-req gd nopass #创建gd机房客户端证书,如果设置密码,客户端连接的时候需要输入密码

==此次操作生成的文件 req: /usr/local/openvpn/key/easyrsa3/pki/reqs/gd.req ==
==此次操作生成的文件 key: /usr/local/openvpn/key/easyrsa3/pki/private/gd.key ==

$ ./easyrsa sign-req client gd #签约客户端证书,需要输入ca密码

==此次操作生成的文件 /usr/local/openvpn/key/easyrsa3/pki/issued/gd.crt ==

$ ./easyrsa sign-req client bj #签约客户端证书,需要输入ca密码

方式2:ca证书不在pki目录下

==此次操作生成的文件 req: /byrd/tools/easy-rsa-master/easyrsa3/pki/reqs/sz.req ==
==此次操作生成的文件 key: /byrd/tools/easy-rsa-master/easyrsa3/pki/private/sz.key ==

./easyrsa sign-req client sz

Easy-RSA error:

Missing expected CA file: index.txt (perhaps you need to run build-ca?)
Run easyrsa without commands for usage and command help.

$ cd /usr/local/openvpn/key/easyrsa3/
$ ./easyrsa import-req /byrd/tools/easy-rsa-master/easyrsa3/pki/reqs/sz.req sz

Note: using Easy-RSA configuration from: ./vars

The request has been successfully imported with a short name of: sz
You may now use this name to perform signing operations on this request.

$ ./easyrsa sign-req client sz

==此次操作生成的文件 /usr/local/openvpn/key/easyrsa3/pki/issued/sz.crt ==

六、OPENVPN所有证书

七、OPENVPN服务端配置

八、OPENVPN客户端配置

九、服务端拨号

服务端:

$ /usr/local/openvpn/sbin/openvpn --config /etc/openvpn/server.conf

客户端(广东):

$ ping 10.8.0.1 #判断拨号状态
PING 10.8.0.1 (10.8.0.1) 56(84) bytes of data.
64 bytes from 10.8.0.1: icmp_seq=1 ttl=64 time=0.613 ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms

VPN段IP拨号验证:
广东IP:10.8.0.10

从vpn客户端机房(广东机房)到VPN服务端机房(杭州机房)互通验证:

到VPN-SERVER(192.168.101.1):

$ route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
10.0.0.0 0.0.0.0 255.0.0.0 U 100 0 0 ens33
10.8.0.0 10.8.0.9 255.255.255.0 UG 0 0 0 tun0
10.8.0.9 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
192.168.102.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37

$ route add -net 192.168.101.0 netmask 255.255.255.0 dev tun0 #广东vpn client配置到187网段的路由,路由接口设备为tun0
#增加到192.168.101.x/24网段的路由,如果不加路由,哪么走默认路由网关1.1.1.2服务器已经被删除路由,因此直接丢包
$ route add -net 192.168.101.0/24 gw 10.8.0.9#广东vpn client配置路由网关是10.8.0.9,其实也是到tun0 和上面任意一条均可

到杭州机房主机(192.168.101.2):

解决方法1:(在192.168.101.2服务器配置到10.8.0.0/24网段的路由)

解决方法2:(在192.168.101.1做IP伪装)

从VPN服务端机房(杭州机房)到vpn客户端机房(广东机房)互通验证:

$ route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
10.0.0.0 0.0.0.0 255.0.0.0 U 100 0 0 ens33
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
10.8.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
192.168.101.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37

$ route add -net 192.168.102.0/24 gw 10.8.0.2
$ route add -net 192.168.103.0/24 gw 10.8.0.2

$ route -n

Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
10.0.0.0 0.0.0.0 255.0.0.0 U 100 0 0 ens33
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
10.8.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
192.168.101.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
192.168.102.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
192.168.103.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0

十、多网互通完整配置:

服务端配置(杭州机房[192.168.101.x/24]):

$ grep -vE "^$|^#|;" /etc/openvpn/server.conf
port 1194
proto tcp
dev tun
ca /etc/openvpn/key/ca.crt
cert /etc/openvpn/key/server.crt
key /etc/openvpn/key/server.key # This file should be kept secret
dh /etc/openvpn/key/dh.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist /tmp/ipp.txt
push "route 192.168.101.0 255.255.255.0"
push "route 192.168.102.0 255.255.255.0"
push "route 192.168.103.0 255.255.255.0"
route 192.168.102.0 255.255.255.0
route 192.168.103.0 255.255.255.0
client-config-dir /etc/openvpn/ccd
client-to-client
keepalive 10 120
cipher AES-256-CBC
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3

$ cat ccd/bj
iroute 192.168.103.0 255.255.255.0
ifconfig-push 10.8.0.14 10.8.0.13
$ cat ccd/gd
iroute 192.168.102.0 255.255.255.0
ifconfig-push 10.8.0.10 10.8.0.9

$ firewall-cmd --zone=public --add-port=1194/tcp

$ firewall-cmd --zone=public --add-masquerade

$ route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
10.0.0.0 0.0.0.0 255.0.0.0 U 100 0 0 ens33
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
10.8.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
192.168.101.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
192.168.102.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
192.168.103.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0

$ ip a
2: ens33: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:3a:e4:cb brd ff:ff:ff:ff:ff:ff
inet 10.4.0.4/8 brd 10.255.255.255 scope global ens33</broadcast,multicast,up,lower_up>

3: ens37: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:3a:e4:d5 brd ff:ff:ff:ff:ff:ff
inet 192.168.101.1/24 brd 192.168.101.255 scope global ens37</broadcast,multicast,up,lower_up>

4: tun0: <pointopoint,multicast,noarp,up,lower_up> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
link/none
inet 10.8.0.1 peer 10.8.0.2/32 scope global tun0
valid_lft forever preferred_lft forever</pointopoint,multicast,noarp,up,lower_up>

$ ping 192.168.102.1 #gd机房 vpn client
PING 192.168.102.1 (192.168.102.1) 56(84) bytes of data.
64 bytes from 192.168.102.1: icmp_seq=1 ttl=64 time=1.98 ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms

$ ping 192.168.102.2 #gd机房 内网
PING 192.168.102.2 (192.168.102.2) 56(84) bytes of data.
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

$ ping 192.168.103.1 #bj机房 vpn client
PING 192.168.103.1 (192.168.103.1) 56(84) bytes of data.
64 bytes from 192.168.103.1: icmp_seq=1 ttl=64 time=1.63 ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms

$ ping 192.168.103.2 #bj机房 内网 开启默认防火墙的情况下(bj机房为centos6.8)
PING 192.168.103.2 (192.168.103.2) 56(84) bytes of data.
From 10.8.0.14 icmp_seq=1 Destination Host Prohibited
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms

$ ping 192.168.103.2 #bj机房 内网 关闭默认防火墙的情况下(bj机房为centos6.8)
PING 192.168.103.2 (192.168.103.2) 56(84) bytes of data.
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

#### 如果vpn client 不做特殊处理,vpn server只能ping通vpn client 所在的局域网IP,无法ping通vpn client内网服务器 ####
centos7:

gd client配置:
方法1:firewall-cmd --zone=public --add-masquerade
方法2:firewall-cmd --direct --passthrough ipv4 -t nat -A POSTROUTING -s 10.8.0.0/24 -o ens37 -j MASQUERADE
方法3:firewall-cmd --direct --passthrough ipv4 -t nat -A POSTROUTING -s 10.8.0.0/24 -o ens37 -j SNAT --to-source 192.168.102.1

gd 内网服务器配置:
方法4:route add -net 10.8.0.0/24 gw 192.168.102.1

$ ping 192.168.102.2 #vpn server验证结果
PING 192.168.102.2 (192.168.102.2) 56(84) bytes of data.
64 bytes from 192.168.102.2: icmp_seq=1 ttl=63 time=42.0 ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms

centos6:

bj client配置:

$ iptables -L FORWARD --line-numbers
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
1 REJECT all -- anywhere anywhere reject-with icmp-host-prohibited

$ iptables -D FORWARD 1

方法1:iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth2 -j SNAT --to-source 192.168.103.1
方法2:iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth2 -j MASQUERADE

客户端配置(广东机房[192.168.102.x/24]):

$ grep -vE "^$|^#|;" /etc/openvpn/client.conf
client
dev tun
proto tcp
remote 10.4.0.4 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/key/ca.crt
cert /etc/openvpn/key/gd.crt
key /etc/openvpn/key/gd.key
remote-cert-tls server
cipher AES-256-CBC
comp-lzo
verb 3

$ route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.2 0.0.0.0 UG 100 0 0 ens33
10.0.0.0 0.0.0.0 255.0.0.0 U 100 0 0 ens33
10.8.0.0 10.8.0.5 255.255.255.0 UG 0 0 0 tun0
10.8.0.5 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
192.168.101.0 10.8.0.5 255.255.255.0 UG 0 0 0 tun0
192.168.102.0 10.8.0.5 255.255.255.0 UG 0 0 0 tun0
192.168.102.0 0.0.0.0 255.255.255.0 U 100 0 0 ens37
192.168.103.0 10.8.0.5 255.255.255.0 UG 0 0 0 tun0

$ ip a
2: ens33: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:ad:e8:37 brd ff:ff:ff:ff:ff:ff
inet 10.4.0.6/8 brd 10.255.255.255 scope global ens33</broadcast,multicast,up,lower_up>

3: ens37: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:ad:e8:41 brd ff:ff:ff:ff:ff:ff
inet 192.168.102.1/24 brd 192.168.102.255 scope global dynamic ens37</broadcast,multicast,up,lower_up>

4: tun0: <pointopoint,multicast,noarp,up,lower_up> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
link/none
inet 10.8.0.10 peer 10.8.0.9/32 scope global tun0
valid_lft forever preferred_lft forever</pointopoint,multicast,noarp,up,lower_up>

客户端配置(北京机房[192.168.103.x/24]):

$ egrep -v "^$|^#|;" /etc/openvpn/client.conf
client
dev tun
proto tcp
remote 10.4.0.4 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/key/ca.crt
cert /etc/openvpn/key/bj.crt
key /etc/openvpn/key/bj.key
remote-cert-tls server
cipher AES-256-CBC
comp-lzo
verb 3

互访验证:

vpn server服务器到其他服务器:

$ ping 192.168.102.1
PING 192.168.102.1 (192.168.102.1) 56(84) bytes of data.
64 bytes from 192.168.102.1: icmp_seq=1 ttl=64 time=1.84 ms

$ ping 192.168.102.2
PING 192.168.102.2 (192.168.102.2) 56(84) bytes of data.
64 bytes from 192.168.102.2: icmp_seq=1 ttl=63 time=16.8 ms

$ ping 192.168.103.1
PING 192.168.103.1 (192.168.103.1) 56(84) bytes of data.
64 bytes from 192.168.103.1: icmp_seq=1 ttl=64 time=1.36 ms

$ ping 192.168.103.2
PING 192.168.103.2 (192.168.103.2) 56(84) bytes of data.
64 bytes from 192.168.103.2: icmp_seq=1 ttl=63 time=2.42 ms

$ tcpdump -nnn -s 10000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 10000 bytes
20:03:42.210040 IP 10.8.0.1 > 192.168.102.1: ICMP echo request, id 1365, seq 1, length 64
20:03:42.211849 IP 192.168.102.1 > 10.8.0.1: ICMP echo reply, id 1365, seq 1, length 64
20:03:44.219057 IP 10.8.0.1 > 192.168.102.2: ICMP echo request, id 1366, seq 1, length 64
20:03:44.222518 IP 192.168.102.2 > 10.8.0.1: ICMP echo reply, id 1366, seq 1, length 64
20:03:50.018020 IP 10.8.0.1 > 192.168.103.1: ICMP echo request, id 1367, seq 1, length 64
20:03:50.020331 IP 192.168.103.1 > 10.8.0.1: ICMP echo reply, id 1367, seq 1, length 64
20:03:52.385603 IP 10.8.0.1 > 192.168.103.2: ICMP echo request, id 1368, seq 1, length 64
20:03:52.387470 IP 192.168.103.2 > 10.8.0.1: ICMP echo reply, id 1368, seq 1, length 64

20:03:42.680819 IP 10.8.0.1 > 192.168.102.1: ICMP echo request, id 1365, seq 1, length 64
20:03:42.680899 IP 192.168.102.1 > 10.8.0.1: ICMP echo reply, id 1365, seq 1, length 64
20:03:44.690503 IP 10.8.0.1 > 192.168.102.2: ICMP echo request, id 1366, seq 1, length 64
20:03:44.691248 IP 192.168.102.2 > 10.8.0.1: ICMP echo reply, id 1366, seq 1, length 64

北京内网服务器(192.168.103.1)到其他服务器:

方法2:
增加默认网关为192.168.102.1 , 哪么所有出网路由走192.168.102.1

备注信息:

Each pair of ifconfig-push addresses represent the virtual client and server IP endpoints. They must be taken from successive /30 subnets in order to be compatible with Windows clients and the TAP-Windows driver. Specifically, the last octet in the IP address of each endpoint pair must be taken from this set:

[ 1, 2] [ 5, 6] [ 9, 10] [ 13, 14] [ 17, 18]
[ 21, 22] [ 25, 26] [ 29, 30] [ 33, 34] [ 37, 38]
[ 41, 42] [ 45, 46] [ 49, 50] [ 53, 54] [ 57, 58]
[ 61, 62] [ 65, 66] [ 69, 70] [ 73, 74] [ 77, 78]
[ 81, 82] [ 85, 86] [ 89, 90] [ 93, 94] [ 97, 98]
[101,102] [105,106] [109,110] [113,114] [117,118]
[121,122] [125,126] [129,130] [133,134] [137,138]
[141,142] [145,146] [149,150] [153,154] [157,158]
[161,162] [165,166] [169,170] [173,174] [177,178]
[181,182] [185,186] [189,190] [193,194] [197,198]
[201,202] [205,206] [209,210] [213,214] [217,218]
[221,222] [225,226] [229,230] [233,234] [237,238]
[241,242] [245,246] [249,250] [253,254]

因为子网内有效的主机数为2^n-2,所以依上面的条件2^2-2=2,即每个子网中实际的主机数为4,子网掩码=256-4=252,使用的又是c类地址,所以该业务所使用的子网掩码的形式为255.255.255.252,可以产生256/4-2=64-2=62个子网,每个子网可用的最大的主机数为2,具体的IP地址如下:
网络 子网掩码 IP地址范围 子网地址 子网广播地址
局域1 255.255.255.252 10.8.0.5~6 10.8.0.4 10.8.0.7
局域2 255.255.255.252 10.8.0.9~10 10.8.0.8 10.8.0.11
局域3 255.255.255.252 10.8.0.13~14 10.8.0.12 10.8.0.15
……
局域62 255.255.255.252 10.8.0.249~250 10.8.0.248 10.8.0.251

十一、证书吊销:

证书吊销:

参考文档:
1:https://openvpn.net/index.php/open-source/documentation/howto.html

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: