k3s环境安装基于Netmaker安装配置

一、基础环境

$ uname -a Linux k3s-master01 6.1.0-28-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux $ uname -a Linux k3s-master02 6.1.0-28-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux $ uname -a Linux k3s-master03 6.8.0-47-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

0 1 2 3 4 5 6 7

$ uname -a Linux k3s-master01 6.1.0-28-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux   $ uname -a Linux k3s-master02 6.1.0-28-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux   $ uname -a Linux k3s-master03 6.8.0-47-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

二、K3s Server配置安装

2.1 必要准备工作

$ wget https://github.com/k3s-io/k3s/releases/download/v1.31.2%2Bk3s1/k3s #--no-check-certificate $ chmod +x k3s && mv k3s /usr/local/bin/ $ wget https://dl.<span class='keyword-inner'><a href="https://note.t4x.org/kubernetes/" title="K8S,kubernetes" target="_blank">K8S</a></span>.io/v1.31.2/<span class='keyword-inner'><a href="https://note.t4x.org/kubernetes/" title="K8S,kubernetes" target="_blank">Kubernetes</a></span>-client-linux-amd64.tar.gz #解压后二进制文件放到/usr/local/bin目录下 $ tar zxf kubernetes-client-linux-amd64.tar.gz $ chmod +x kubernetes/client/bin/* $ mv kubernetes/client/bin/* /usr/local/bin/ $ wget https://github.com/k3s-io/helm-controller/releases/download/v0.16.5/helm-controller-amd64 $ chmod +x helm-controller-amd64 $ mv helm-controller-amd64 /usr/local/bin/helm

0 1 2 3 4 5 6 7 8

$ wget https://github.com/k3s-io/k3s/releases/download/v1.31.2%2Bk3s1/k3s #--no-check-certificate $ chmod +x k3s && mv k3s /usr/local/bin/ $ wget https://dl.k8s.io/v1.31.2/kubernetes-client-linux-amd64.tar.gz    #解压后二进制文件放到/usr/local/bin目录下 $ tar zxf kubernetes-client-linux-amd64.tar.gz $ chmod +x  kubernetes/client/bin/* $ mv kubernetes/client/bin/* /usr/local/bin/ $ wget https://github.com/k3s-io/helm-controller/releases/download/v0.16.5/helm-controller-amd64 $ chmod +x helm-controller-amd64 $ mv helm-controller-amd64 /usr/local/bin/helm

2.2 k3s-master01启动文件

$ cat /etc/systemd/system/k3s.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io Wants=network-online.target [Install] WantedBy=multi-user.target [Service] Type=notify EnvironmentFile=-/etc/default/%N EnvironmentFile=-/etc/sysconfig/%N EnvironmentFile=-/etc/systemd/system/k3s.service.env KillMode=process Delegate=yes # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=1048576 LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always RestartSec=5s ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/k3s \ server \ --bind-address '0.0.0.0' \ --write-kubeconfig-mode '0600' \ --write-kubeconfig '/root/.kube/config' \ --data-dir '/data/k3s' \ --cluster-cidr 10.42.0.0/16 \ --service-cidr 10.43.0.0/16 \ --service-node-port-range '30000-42767' \ --tls-san 'k3s.t4x.org' \ --node-ip '本地IP' \ #基于wireguard所以写了wireguard的IP --advertise-address 'wg' \ --node-external-ip 'wg' \ --node-label 'nodetype=server' \ --secrets-encryption \ #--flannel-backend 'vxlan' \ --disable-network-policy \ --flannel-iface 'netmaker' \ # 因为我使用的Netmaker和wireguard已经搭建好了wg网络,直接调用网卡接口 --flannel-backend 'none' \ --disable 'servicelb' \ --disable 'traefik' \ --log '/var/log/k3s.log' \ --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \ --kube-proxy-arg "metrics-bind-address=0.0.0.0" \ --datastore-endpoint '<span class='keyword-inner'><a href="https://note.t4x.org/database/" title="database,mysql" target="_blank">MySQL</a></span>://user:password@tcp(1.1.1.1:3306)/dbname'

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

$ cat /etc/systemd/system/k3s.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io Wants=network-online.target   [Install] WantedBy=multi-user.target   [Service] Type=notify EnvironmentFile=-/etc/default/%N EnvironmentFile=-/etc/sysconfig/%N EnvironmentFile=-/etc/systemd/system/k3s.service.env KillMode=process Delegate=yes # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=1048576 LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always RestartSec=5s ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/k3s \     server \     --bind-address '0.0.0.0' \     --write-kubeconfig-mode '0600' \     --write-kubeconfig  '/root/.kube/config' \     --data-dir '/data/k3s' \     --cluster-cidr 10.42.0.0/16 \     --service-cidr 10.43.0.0/16 \     --service-node-port-range '30000-42767' \     --tls-san 'k3s.t4x.org' \     --node-ip '本地IP' \    #基于wireguard所以写了wireguard的IP     --advertise-address  'wg' \     --node-external-ip 'wg' \     --node-label 'nodetype=server' \     --secrets-encryption \     #--flannel-backend 'vxlan' \     --disable-network-policy \     --flannel-iface 'netmaker' \    # 因为我使用的Netmaker和wireguard已经搭建好了wg网络,直接调用网卡接口     --flannel-backend 'none' \     --disable 'servicelb' \     --disable 'traefik' \     --log '/var/log/k3s.log' \     --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \     --kube-proxy-arg "metrics-bind-address=0.0.0.0" \     --datastore-endpoint 'mysql://user:password@tcp(1.1.1.1:3306)/dbname'

2.3 k3s-masterN启动文件 只是增加了master01的token

$ cat /etc/systemd/system/k3s.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io Wants=network-online.target [Install] WantedBy=multi-user.target [Service] Type=notify EnvironmentFile=-/etc/default/%N EnvironmentFile=-/etc/sysconfig/%N EnvironmentFile=-/etc/systemd/system/k3s.service.env KillMode=process Delegate=yes # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=1048576 LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always RestartSec=5s ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/k3s \ server \ --bind-address '0.0.0.0' \ --write-kubeconfig-mode '0600' \ --write-kubeconfig '/root/.kube/config' \ --data-dir '/data/k3s' \ --cluster-cidr 10.42.0.0/16 \ --service-cidr 10.43.0.0/16 \ --service-node-port-range '30000-42767' \ --tls-san 'k3s.t4x.org' \ --node-ip 'wg ip' \ --node-external-ip '公网IP' \ --advertise-address 'wg ip' \ --node-label 'nodetype=worker' \ --secrets-encryption \ --flannel-iface 'netmaker' \ --flannel-backend 'none' \ --disable 'servicelb' \ --disable 'traefik' \ --log '/var/log/k3s.log' \ --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \ --kube-proxy-arg "metrics-bind-address=0.0.0.0" \ --token 'K***::server:***' \ --datastore-endpoint 'mysql://user:password@tcp(1.1.1.1:3306)/dbname'

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

$ cat /etc/systemd/system/k3s.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io Wants=network-online.target   [Install] WantedBy=multi-user.target   [Service] Type=notify EnvironmentFile=-/etc/default/%N EnvironmentFile=-/etc/sysconfig/%N EnvironmentFile=-/etc/systemd/system/k3s.service.env KillMode=process Delegate=yes # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=1048576 LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always RestartSec=5s ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/k3s \     server \     --bind-address '0.0.0.0' \     --write-kubeconfig-mode '0600' \     --write-kubeconfig  '/root/.kube/config' \     --data-dir '/data/k3s' \     --cluster-cidr 10.42.0.0/16 \     --service-cidr 10.43.0.0/16 \     --service-node-port-range '30000-42767' \     --tls-san 'k3s.t4x.org' \     --node-ip 'wg ip' \     --node-external-ip '公网IP' \     --advertise-address  'wg ip' \     --node-label 'nodetype=worker' \     --secrets-encryption \     --flannel-iface 'netmaker' \     --flannel-backend 'none' \     --disable 'servicelb' \     --disable 'traefik' \     --log '/var/log/k3s.log' \     --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \     --kube-proxy-arg "metrics-bind-address=0.0.0.0" \     --token 'K***::server:***' \     --datastore-endpoint 'mysql://user:password@tcp(1.1.1.1:3306)/dbname'

类似于: INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh --node-external-ip 82.157.xx.xx --advertise-address 82.157.xx.xx --node-ip 10.10.10.1 --flannel-iface wg0 --kube-proxy-arg "proxy-mode=ipvs" 类似于: INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh --node-external-ip 82.157.xx.xx --advertise-address 82.157.xx.xx --flannel-backend wireguard --kube-proxy-arg "proxy-mode=ipvs"

0 1

类似于: INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh --node-external-ip 82.157.xx.xx --advertise-address  82.157.xx.xx --node-ip 10.10.10.1 --flannel-iface wg0 --kube-proxy-arg "proxy-mode=ipvs" 类似于: INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh --node-external-ip 82.157.xx.xx --advertise-address  82.157.xx.xx  --flannel-backend wireguard --kube-proxy-arg "proxy-mode=ipvs"

SourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/

2.4 部署完成后的状态

三、K3s Server配置验证

3.1 2台master状态

$ kubectl get nodes,services,pods --all-namespaces -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node/k3s-master03 Ready control-plane,master 42m v1.31.2+k3s1 WireGuard IP 公网ip Ubuntu 24.04.1 LTS 6.8.0-47-generic containerd://1.7.22-k3s1 node/master-internal Ready control-plane,master 73m v1.31.2+k3s1 WireGuard IP WireGuard IP Debian GNU/Linux 12 (bookworm) 6.1.0-28-amd64 containerd://1.7.22-k3s1 NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 73m <none> kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 73m k8s-app=kube-dns kube-system service/metrics-server ClusterIP 10.43.172.21 <none> 443/TCP 73m k8s-app=metrics-server NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system pod/calico-kube-controllers-5d7d9cdfd8-nsf4h 1/1 Running 0 52m 10.42.48.193 master-internal <none> <none> kube-system pod/calico-node-575n2 1/1 Running 0 42m WireGuard IP k3s-master03 <none> <none> kube-system pod/calico-node-phbzv 1/1 Running 0 52m WireGuard IP master-internal <none> <none> kube-system pod/coredns-56f6fc8fd7-9vgjh 1/1 Running 1 (61m ago) 73m 10.42.48.197 master-internal <none> <none> kube-system pod/local-path-provisioner-5cf85fd84d-5l9tm 1/1 Running 1 (61m ago) 73m 10.42.48.200 master-internal <none> <none> kube-system pod/metrics-server-5985cbc9d7-zfh7f 1/1 Running 1 (61m ago) 73m 10.42.48.198 master-internal <none> <none>

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

$ kubectl get nodes,services,pods --all-namespaces -o wide NAME                   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME node/k3s-master03      Ready    control-plane,master   42m   v1.31.2+k3s1   WireGuard IP  公网ip          Ubuntu 24.04.1 LTS               6.8.0-47-generic   containerd://1.7.22-k3s1 node/master-internal   Ready    control-plane,master   73m   v1.31.2+k3s1   WireGuard IP  WireGuard IP    Debian GNU/Linux 12 (bookworm)   6.1.0-28-amd64     containerd://1.7.22-k3s1   NAMESPACE     NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE   SELECTOR default       service/kubernetes       ClusterIP   10.43.0.1      <none>        443/TCP                  73m   <none> kube-system   service/kube-dns         ClusterIP   10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP   73m   k8s-app=kube-dns kube-system   service/metrics-server   ClusterIP   10.43.172.21   <none>        443/TCP                  73m   k8s-app=metrics-server   NAMESPACE     NAME                                           READY   STATUS    RESTARTS      AGE   IP             NODE              NOMINATED NODE   READINESS GATES kube-system   pod/calico-kube-controllers-5d7d9cdfd8-nsf4h   1/1     Running   0             52m   10.42.48.193   master-internal   <none>           <none> kube-system   pod/calico-node-575n2                          1/1     Running   0             42m   WireGuard IP   k3s-master03      <none>           <none> kube-system   pod/calico-node-phbzv                          1/1     Running   0             52m   WireGuard IP   master-internal   <none>           <none> kube-system   pod/coredns-56f6fc8fd7-9vgjh                   1/1     Running   1 (61m ago)   73m   10.42.48.197   master-internal   <none>           <none> kube-system   pod/local-path-provisioner-5cf85fd84d-5l9tm    1/1     Running   1 (61m ago)   73m   10.42.48.200   master-internal   <none>           <none> kube-system   pod/metrics-server-5985cbc9d7-zfh7f            1/1     Running   1 (61m ago)   73m   10.42.48.198   master-internal   <none>           <none>

3.2 路由以及服务器当前IP

$ route -n #master-internal服务器 Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.42.48.192 0.0.0.0 255.255.255.192 U 0 0 0 * 10.42.48.193 0.0.0.0 255.255.255.255 UH 0 0 0 califf56c4d7311 10.42.48.197 0.0.0.0 255.255.255.255 UH 0 0 0 cali1592a4fd865 10.42.48.198 0.0.0.0 255.255.255.255 UH 0 0 0 cali3839105c1fb 10.42.48.200 0.0.0.0 255.255.255.255 UH 0 0 0 calicc726754017 10.42.66.128 对端WireGuardIP 255.255.255.192 UG 0 0 0 tunl0 $ ip a #master-internal服务器 5: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet x.x.x.x/16 brd x.x.255.255 scope global netmaker valid_lft forever preferred_lft forever 6: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether 12:bb:a4:29:a3:92 brd ff:ff:ff:ff:ff:ff inet 10.43.0.10/32 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.43.172.21/32 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.43.0.1/32 scope global kube-ipvs0 valid_lft forever preferred_lft forever 7: cali1592a4fd865@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000 link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c8f849e9-6c89-064e-59b6-cf991adb1a03 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever 8: cali3839105c1fb@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000 link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c0cc0621-9940-9940-a2bc-f2143acc8a6a inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever 10: calicc726754017@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000 link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-999cbcb6-e620-a305-bff8-1e6d03d0ccaf inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever 11: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.42.48.192/32 scope global tunl0 valid_lft forever preferred_lft forever 16: califf56c4d7311@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000 link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-7707ae19-c28a-a507-8c20-202b6a7cfa64 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever $ route -n #k3s-master03服务器 Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.42.48.192 对端WireGuardIP 255.255.255.192 UG 0 0 0 tunl0 10.42.66.128 0.0.0.0 255.255.255.192 U 0 0 0 * $ ip a #k3s-master03服务器 3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.42.66.128/32 scope global tunl0 valid_lft forever preferred_lft forever 5: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet x.x.x.x/16 brd x.x.255.255 scope global netmaker valid_lft forever preferred_lft forever 6: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether aa:10:27:4e:b4:79 brd ff:ff:ff:ff:ff:ff inet 10.43.0.10/32 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.43.172.21/32 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.43.0.1/32 scope global kube-ipvs0 valid_lft forever preferred_lft forever

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

$ route -n    #master-internal服务器 Kernel IP routing table Destination     Gateway         Genmask         Flags Metric Ref    Use Iface 10.42.48.192    0.0.0.0         255.255.255.192 U     0      0        0 * 10.42.48.193    0.0.0.0         255.255.255.255 UH    0      0        0 califf56c4d7311 10.42.48.197    0.0.0.0         255.255.255.255 UH    0      0        0 cali1592a4fd865 10.42.48.198    0.0.0.0         255.255.255.255 UH    0      0        0 cali3839105c1fb 10.42.48.200    0.0.0.0         255.255.255.255 UH    0      0        0 calicc726754017 10.42.66.128    对端WireGuardIP 255.255.255.192 UG    0      0        0 tunl0   $ ip a    #master-internal服务器 5: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000     link/none     inet x.x.x.x/16 brd x.x.255.255 scope global netmaker        valid_lft forever preferred_lft forever 6: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default     link/ether 12:bb:a4:29:a3:92 brd ff:ff:ff:ff:ff:ff     inet 10.43.0.10/32 scope global kube-ipvs0        valid_lft forever preferred_lft forever     inet 10.43.172.21/32 scope global kube-ipvs0        valid_lft forever preferred_lft forever     inet 10.43.0.1/32 scope global kube-ipvs0        valid_lft forever preferred_lft forever 7: cali1592a4fd865@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000     link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c8f849e9-6c89-064e-59b6-cf991adb1a03     inet6 fe80::ecee:eeff:feee:eeee/64 scope link        valid_lft forever preferred_lft forever 8: cali3839105c1fb@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000     link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c0cc0621-9940-9940-a2bc-f2143acc8a6a     inet6 fe80::ecee:eeff:feee:eeee/64 scope link        valid_lft forever preferred_lft forever 10: calicc726754017@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000     link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-999cbcb6-e620-a305-bff8-1e6d03d0ccaf     inet6 fe80::ecee:eeff:feee:eeee/64 scope link        valid_lft forever preferred_lft forever 11: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000     link/ipip 0.0.0.0 brd 0.0.0.0     inet 10.42.48.192/32 scope global tunl0        valid_lft forever preferred_lft forever 16: califf56c4d7311@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000     link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-7707ae19-c28a-a507-8c20-202b6a7cfa64     inet6 fe80::ecee:eeff:feee:eeee/64 scope link        valid_lft forever preferred_lft forever   $ route -n    #k3s-master03服务器 Kernel IP routing table Destination     Gateway         Genmask         Flags Metric Ref    Use Iface 10.42.48.192    对端WireGuardIP 255.255.255.192 UG    0      0        0 tunl0 10.42.66.128    0.0.0.0         255.255.255.192 U     0      0        0 *   $ ip a    #k3s-master03服务器 3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000     link/ipip 0.0.0.0 brd 0.0.0.0     inet 10.42.66.128/32 scope global tunl0        valid_lft forever preferred_lft forever 5: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000     link/none     inet x.x.x.x/16 brd x.x.255.255 scope global netmaker        valid_lft forever preferred_lft forever 6: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default     link/ether aa:10:27:4e:b4:79 brd ff:ff:ff:ff:ff:ff     inet 10.43.0.10/32 scope global kube-ipvs0        valid_lft forever preferred_lft forever     inet 10.43.172.21/32 scope global kube-ipvs0        valid_lft forever preferred_lft forever     inet 10.43.0.1/32 scope global kube-ipvs0        valid_lft forever preferred_lft forever

3.3 连通性测试

$ curl 'https://10.42.48.198:10250' -k #k3s-master03服务器操作 { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"", "reason": "Forbidden", "details": {}, "code": 403 } $ telnet 10.43.0.1 443 Trying 10.43.0.1... Connected to 10.43.0.1. Escape character is '^]'. $ telnet 10.43.0.10 53 Trying 10.43.0.10... Connected to 10.43.0.10. Escape character is '^]'. $ ping 10.42.48.197 PING 10.42.48.197 (10.42.48.197) 56(84) bytes of data. 64 bytes from 10.42.48.197: icmp_seq=1 ttl=63 time=29.5 ms $ telnet 10.43.172.21 443 #master-internal服务器 Trying 10.43.172.21... Connected to 10.43.172.21. Escape character is '^]'. telnet> quit Connection closed. $ telnet 10.43.0.10 53 #master-internal服务器 Trying 10.43.0.10... Connected to 10.43.0.10. Escape character is '^]'.

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

$ curl 'https://10.42.48.198:10250' -k    #k3s-master03服务器操作 {   "kind": "Status",   "apiVersion": "v1",   "metadata": {},   "status": "Failure",   "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",   "reason": "Forbidden",   "details": {},   "code": 403 }   $ telnet 10.43.0.1 443 Trying 10.43.0.1... Connected to 10.43.0.1. Escape character is '^]'.   $ telnet 10.43.0.10 53 Trying 10.43.0.10... Connected to 10.43.0.10. Escape character is '^]'.   $ ping 10.42.48.197 PING 10.42.48.197 (10.42.48.197) 56(84) bytes of data. 64 bytes from 10.42.48.197: icmp_seq=1 ttl=63 time=29.5 ms     $ telnet 10.43.172.21 443    #master-internal服务器 Trying 10.43.172.21... Connected to 10.43.172.21. Escape character is '^]'.   telnet> quit Connection closed. $ telnet 10.43.0.10 53    #master-internal服务器 Trying 10.43.0.10... Connected to 10.43.0.10. Escape character is '^]'.

calico-node-xxxx 其实就是wg的IP地址,因为基于wg,是前提互通的前提,无测试必要!!!

0	calico-node-xxxx 其实就是wg的IP地址,因为基于wg,是前提互通的前提,无测试必要!!!

SourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/

四、K3s Agent配置安装

$ cat /etc/systemd/system/k3s-agent.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io Wants=network-online.target [Install] WantedBy=multi-user.target [Service] Type=exec EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env KillMode=process Delegate=yes LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always RestartSec=5s ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/k3s agent \ --node-label 'nodetype=worker' \ --node-external-ip '公网IP' \ --node-ip 'WG IP' \ --data-dir '/data/k3s' \ --token 'XXX' \ --log '/var/log/k3s_agent.log' \ --server 'https://X.X.X.X:6443' \ --disable-network-policy \ --flannel-iface 'netmaker' \ --flannel-backend 'none' \ --node-name 'blog' \ --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \ --kube-proxy-arg "metrics-bind-address=0.0.0.0"

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

$ cat /etc/systemd/system/k3s-agent.service [Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io Wants=network-online.target   [Install] WantedBy=multi-user.target   [Service] Type=exec EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env KillMode=process Delegate=yes LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always RestartSec=5s ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/k3s agent \     --node-label 'nodetype=worker' \     --node-external-ip '公网IP' \     --node-ip 'WG IP' \     --data-dir '/data/k3s' \     --token 'XXX' \     --log '/var/log/k3s_agent.log' \     --server 'https://X.X.X.X:6443' \     --disable-network-policy \     --flannel-iface 'netmaker' \     --flannel-backend 'none' \     --node-name 'blog' \     --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \     --kube-proxy-arg "metrics-bind-address=0.0.0.0"

SourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/

五、k3s最终状态确认

$ kubectl label node hz-blog node-role.kubernetes.io/worker=worker $ kubectl label node bj-proxy node-role.kubernetes.io/worker=worker $ kubectl label node sh-git node-role.kubernetes.io/worker=worker $ kubectl get node,pods,svc --all-namespaces -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node/bj-proxy Ready worker 5m19s v1.31.2+k3s1 node-ip(wg) 公网IP CentOS Stream 9 5.14.0-444.el9.x86_64 containerd://1.7.22-k3s1 node/hz-blog Ready worker 15h v1.31.2+k3s1 node-ip(wg) 公网IP CentOS Stream 9 5.14.0-407.el9.x86_64 containerd://1.7.22-k3s1 node/k3s-master03 Ready control-plane,master 37h v1.31.2+k3s1 node-ip(wg) 公网IP Ubuntu 24.04.1 LTS 6.8.0-47-generic containerd://1.7.22-k3s1 node/master-internal Ready control-plane,master 38h v1.31.2+k3s1 node-ip(wg) 本地无公网用wgip Debian GNU/Linux 12 (bookworm) 6.1.0-28-amd64 containerd://1.7.22-k3s1 node/sh-git NotReady worker 3m9s v1.31.2+k3s1 node-ip(wg) 公网IP CentOS Stream 9 5.14.0-407.el9.x86_64 containerd://1.7.22-k3s1 NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system pod/calico-kube-controllers-5d7d9cdfd8-nsf4h 1/1 Running 0 37h 10.42.48.193 master-internal <none> <none> kube-system pod/calico-node-575n2 1/1 Running 0 37h node-ip(wg) k3s-master03 <none> <none> kube-system pod/calico-node-b87g9 1/1 Running 0 5m18s node-ip(wg) bj-proxy <none> <none> kube-system pod/calico-node-dv76v 1/1 Running 0 3m9s node-ip(wg) sh-git <none> <none> kube-system pod/calico-node-phbzv 1/1 Running 0 37h node-ip(wg) master-internal <none> <none> kube-system pod/calico-node-wkw9g 1/1 Running 0 15h node-ip(wg) hz-blog <none> <none> kube-system pod/coredns-56f6fc8fd7-9vgjh 1/1 Running 1 (37h ago) 38h 10.42.48.197 master-internal <none> <none> kube-system pod/local-path-provisioner-5cf85fd84d-5l9tm 1/1 Running 1 (37h ago) 38h 10.42.48.200 master-internal <none> <none> kube-system pod/metrics-server-5985cbc9d7-zfh7f 1/1 Running 1 (37h ago) 38h 10.42.48.198 master-internal <none> <none> NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 38h <none> kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 38h k8s-app=kube-dns kube-system service/metrics-server ClusterIP 10.43.172.21 <none> 443/TCP 38h k8s-app=metrics-server $ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% bj-proxy 98m 4% 2192Mi 59% hz-blog 71m 3% 2451Mi 68% k3s-master03 139m 6% 1134Mi 68% master-internal 429m 10% 1748Mi 44% sh-git 82m 4% 1110Mi 66%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

$ kubectl label node hz-blog node-role.kubernetes.io/worker=worker $ kubectl label node bj-proxy node-role.kubernetes.io/worker=worker $ kubectl label node sh-git node-role.kubernetes.io/worker=worker $ kubectl get node,pods,svc --all-namespaces -owide NAME                   STATUS     ROLES                  AGE     VERSION        INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                         KERNEL-VERSION          CONTAINER-RUNTIME node/bj-proxy          Ready      worker                 5m19s   v1.31.2+k3s1   node-ip(wg)   公网IP           CentOS Stream 9                  5.14.0-444.el9.x86_64   containerd://1.7.22-k3s1 node/hz-blog           Ready      worker                 15h     v1.31.2+k3s1   node-ip(wg)   公网IP           CentOS Stream 9                  5.14.0-407.el9.x86_64   containerd://1.7.22-k3s1 node/k3s-master03      Ready      control-plane,master   37h     v1.31.2+k3s1   node-ip(wg)   公网IP           Ubuntu 24.04.1 LTS               6.8.0-47-generic        containerd://1.7.22-k3s1 node/master-internal   Ready      control-plane,master   38h     v1.31.2+k3s1   node-ip(wg)   本地无公网用wgip  Debian GNU/Linux 12 (bookworm)   6.1.0-28-amd64          containerd://1.7.22-k3s1 node/sh-git            NotReady   worker                 3m9s    v1.31.2+k3s1   node-ip(wg)   公网IP           CentOS Stream 9                  5.14.0-407.el9.x86_64   containerd://1.7.22-k3s1   NAMESPACE     NAME                                           READY   STATUS    RESTARTS      AGE     IP             NODE              NOMINATED NODE   READINESS GATES kube-system   pod/calico-kube-controllers-5d7d9cdfd8-nsf4h   1/1     Running   0             37h     10.42.48.193   master-internal   <none>           <none> kube-system   pod/calico-node-575n2                          1/1     Running   0             37h     node-ip(wg)    k3s-master03      <none>           <none> kube-system   pod/calico-node-b87g9                          1/1     Running   0             5m18s   node-ip(wg)    bj-proxy          <none>           <none> kube-system   pod/calico-node-dv76v                          1/1     Running   0             3m9s    node-ip(wg)    sh-git            <none>           <none> kube-system   pod/calico-node-phbzv                          1/1     Running   0             37h     node-ip(wg)    master-internal   <none>           <none> kube-system   pod/calico-node-wkw9g                          1/1     Running   0             15h     node-ip(wg)    hz-blog           <none>           <none> kube-system   pod/coredns-56f6fc8fd7-9vgjh                   1/1     Running   1 (37h ago)   38h     10.42.48.197   master-internal   <none>           <none> kube-system   pod/local-path-provisioner-5cf85fd84d-5l9tm    1/1     Running   1 (37h ago)   38h     10.42.48.200   master-internal   <none>           <none> kube-system   pod/metrics-server-5985cbc9d7-zfh7f            1/1     Running   1 (37h ago)   38h     10.42.48.198   master-internal   <none>           <none>   NAMESPACE     NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE   SELECTOR default       service/kubernetes       ClusterIP   10.43.0.1      <none>        443/TCP                  38h   <none> kube-system   service/kube-dns         ClusterIP   10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP   38h   k8s-app=kube-dns kube-system   service/metrics-server   ClusterIP   10.43.172.21   <none>        443/TCP                  38h   k8s-app=metrics-server   $ kubectl top nodes NAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   bj-proxy          98m          4%     2192Mi          59%       hz-blog           71m          3%     2451Mi          68%       k3s-master03      139m         6%     1134Mi          68%       master-internal   429m         10%    1748Mi          44%       sh-git            82m          4%     1110Mi          66%

SourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/

六、K3s worker节点pod连通性测试

$ kubectl taint node master-internal nodetype=master:NoSchedule $ cat nginx.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 4 # 指定Pod副本数为3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest # 使用最新的Nginx镜像 ports: - containerPort: 80 # Nginx监听的端口 $ kubectl create -f nginx.yaml deployment.apps/nginx-deployment created $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-54b9c68f67-8rbts 1/1 Running 0 103s 10.42.222.129 sh-git <none> <none> nginx-deployment-54b9c68f67-d4tmq 1/1 Running 0 102s 10.42.66.147 k3s-master03 <none> <none> nginx-deployment-54b9c68f67-d62t2 1/1 Running 0 103s 10.42.174.205 hz-blog <none> <none> nginx-deployment-54b9c68f67-ndbk6 1/1 Running 0 103s 10.42.201.1 bj-proxy <none> <none> $ ping 10.42.222.129 #master-internal ping 测试 PING 10.42.222.129 (10.42.222.129) 56(84) bytes of data. 64 bytes from 10.42.222.129: icmp_seq=1 ttl=63 time=31.5 ms ^C $ ping 10.42.66.147 PING 10.42.66.147 (10.42.66.147) 56(84) bytes of data. 64 bytes from 10.42.66.147: icmp_seq=1 ttl=63 time=29.8 ms ^C $ ping 10.42.174.205 PING 10.42.174.205 (10.42.174.205) 56(84) bytes of data. 64 bytes from 10.42.174.205: icmp_seq=1 ttl=63 time=32.6 ms 64 bytes from 10.42.174.205: icmp_seq=2 ttl=63 time=32.6 ms ^C $ ping 10.42.201.1 PING 10.42.201.1 (10.42.201.1) 56(84) bytes of data. 64 bytes from 10.42.201.1: icmp_seq=1 ttl=63 time=26.2 ms ^C $ ping 10.42.222.129 #sh-git ping 测试 PING 10.42.222.129 (10.42.222.129) 56(84) bytes of data. 64 bytes from 10.42.222.129: icmp_seq=1 ttl=64 time=0.245 ms ^C $ ping 10.42.66.147 PING 10.42.66.147 (10.42.66.147) 56(84) bytes of data. 64 bytes from 10.42.66.147: icmp_seq=1 ttl=63 time=27.0 ms ^C $ ping 10.42.174.205 PING 10.42.174.205 (10.42.174.205) 56(84) bytes of data. 64 bytes from 10.42.174.205: icmp_seq=1 ttl=63 time=9.47 ms ^C $ ping 10.42.201.1 PING 10.42.201.1 (10.42.201.1) 56(84) bytes of data. 64 bytes from 10.42.201.1: icmp_seq=1 ttl=63 time=34.9 ms ^C $ kubectl get services -o wide --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 38h <none> kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 38h k8s-app=kube-dns kube-system metrics-server ClusterIP 10.43.172.21 <none> 443/TCP 38h k8s-app=metrics-server $ telnet 10.43.0.1 443 #bj-proxy 到servicestelnet测试 Trying 10.43.0.1... Connected to 10.43.0.1. Escape character is '^]'. $ telnet 10.43.0.10 53 Trying 10.43.0.10... Connected to 10.43.0.10. Escape character is '^]'. $ telnet 10.43.172.21 443 Trying 10.43.172.21... Connected to 10.43.172.21. Escape character is '^]'.

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

$ kubectl taint node master-internal nodetype=master:NoSchedule $ cat nginx.yaml apiVersion: apps/v1 kind: Deployment metadata:   name: nginx-deployment spec:   replicas: 4  # 指定Pod副本数为3   selector:     matchLabels:       app: nginx   template:     metadata:       labels:         app: nginx     spec:       containers:       - name: nginx         image: nginx:latest  # 使用最新的Nginx镜像         ports:         - containerPort: 80  # Nginx监听的端口 $ kubectl create -f nginx.yaml       deployment.apps/nginx-deployment created $ kubectl get pods -o wide     NAME                                READY   STATUS    RESTARTS   AGE   IP          NODE             NOMINATED NODE   READINESS GATES NAME                                READY   STATUS    RESTARTS   AGE    IP              NODE           NOMINATED NODE   READINESS GATES nginx-deployment-54b9c68f67-8rbts   1/1     Running   0          103s   10.42.222.129   sh-git         <none>           <none> nginx-deployment-54b9c68f67-d4tmq   1/1     Running   0          102s   10.42.66.147    k3s-master03   <none>           <none> nginx-deployment-54b9c68f67-d62t2   1/1     Running   0          103s   10.42.174.205   hz-blog        <none>           <none> nginx-deployment-54b9c68f67-ndbk6   1/1     Running   0          103s   10.42.201.1     bj-proxy       <none>           <none>   $ ping 10.42.222.129    #master-internal ping 测试 PING 10.42.222.129 (10.42.222.129) 56(84) bytes of data. 64 bytes from 10.42.222.129: icmp_seq=1 ttl=63 time=31.5 ms ^C $ ping 10.42.66.147 PING 10.42.66.147 (10.42.66.147) 56(84) bytes of data. 64 bytes from 10.42.66.147: icmp_seq=1 ttl=63 time=29.8 ms ^C $ ping 10.42.174.205 PING 10.42.174.205 (10.42.174.205) 56(84) bytes of data. 64 bytes from 10.42.174.205: icmp_seq=1 ttl=63 time=32.6 ms 64 bytes from 10.42.174.205: icmp_seq=2 ttl=63 time=32.6 ms ^C $ ping 10.42.201.1 PING 10.42.201.1 (10.42.201.1) 56(84) bytes of data. 64 bytes from 10.42.201.1: icmp_seq=1 ttl=63 time=26.2 ms ^C $ ping 10.42.222.129    #sh-git ping 测试 PING 10.42.222.129 (10.42.222.129) 56(84) bytes of data. 64 bytes from 10.42.222.129: icmp_seq=1 ttl=64 time=0.245 ms ^C $ ping 10.42.66.147 PING 10.42.66.147 (10.42.66.147) 56(84) bytes of data. 64 bytes from 10.42.66.147: icmp_seq=1 ttl=63 time=27.0 ms ^C $ ping 10.42.174.205 PING 10.42.174.205 (10.42.174.205) 56(84) bytes of data. 64 bytes from 10.42.174.205: icmp_seq=1 ttl=63 time=9.47 ms ^C $ ping 10.42.201.1 PING 10.42.201.1 (10.42.201.1) 56(84) bytes of data. 64 bytes from 10.42.201.1: icmp_seq=1 ttl=63 time=34.9 ms ^C $ kubectl get services -o wide --all-namespaces NAMESPACE     NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE   SELECTOR default       kubernetes       ClusterIP   10.43.0.1      <none>        443/TCP                  38h   <none> kube-system   kube-dns         ClusterIP   10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP   38h   k8s-app=kube-dns kube-system   metrics-server   ClusterIP   10.43.172.21   <none>        443/TCP                  38h   k8s-app=metrics-server   $ telnet 10.43.0.1 443    #bj-proxy 到servicestelnet测试 Trying 10.43.0.1... Connected to 10.43.0.1. Escape character is '^]'. $ telnet 10.43.0.10 53 Trying 10.43.0.10... Connected to 10.43.0.10. Escape character is '^]'. $ telnet 10.43.172.21 443 Trying 10.43.172.21... Connected to 10.43.172.21. Escape character is '^]'.

SourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/

七、系统日志

九、问题解决

Q1：

$ kubectl get cs E1130 17:57:55.274901 11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.279022 11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.281379 11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.283712 11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.286483 11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" The connection to the server localhost:8080 was refused - did you specify the right host or port?

0 1 2 3 4 5 6

$ kubectl get cs E1130 17:57:55.274901   11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.279022   11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.281379   11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.283712   11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" E1130 17:57:55.286483   11722 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp [::1]:8080: connect: connection refused" The connection to the server localhost:8080 was refused - did you specify the right host or port?

A1:

cat <<EOF >> /root/.bashrc export KUBECONFIG=/etc/rancher/k3s/k3s.yaml EOF

0 1 2

cat <<EOF >> /root/.bashrc export KUBECONFIG=/etc/rancher/k3s/k3s.yaml EOF

SourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/

参考文档：
1：https://docs.k3s.io/zh/
2：https://docs.tigera.io/calico/latest/getting-started/kubernetes/k3s/quickstart
3：https://github.com/containerd/nerdctl
4：https://note.t4x.org/basic/cross-wireguard-config/
5：https://note.t4x.org/basic/netmaker-manager-wireguard/
6：https://docs.tigera.io/archive/v3.18/networking/ip-autodetection
7：https://github.com/chobits/ngx_http_proxy_connect_module
8：https://blog.csdn.net/wq1205750492/article/details/124883196
9：https://developer.aliyun.com/mirror/docker-ce
10：https://baijiahao.baidu.com/s?id=1685684533566281761
11：https://github.com/k3s-io/k3s/
12：https://docs.rancher.cn/docs/k3s/architecture/_index
13：https://ranchermanager.docs.rancher.com/zh/v2.6/reference-guides/kubernetes-conceptsSourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/ SourceByrd's Weblog-https://note.t4x.org/kubernetes/k3s-server-agent-config/