本站基于K3S的网络架构拓扑

一、基础环境

$ uname -a Linux jp-master01 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux $ uname -a Linux tw-master02 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux $ uname -a Linux hk-master03 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux $ uname -a Linux singapore-worker 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux

0 1 2 3 4 5 6 7 8 9 10

$ uname -a Linux jp-master01 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux   $ uname -a Linux tw-master02 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux   $ uname -a Linux hk-master03 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux   $ uname -a Linux singapore-worker 6.1.0-28-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux

环境基于GCP三个月免费试用搭建!

二、架构拓扑

三、搭建过程

3.1 Netmaker搭建-基于docker compose

1、将域名api.jp.t4x.org、dashboard.jp.t4x.org、broker.jp.t4x.org解析到服务器IP #比如本次实验的IP地址:35.213.98.66
2、允许TCP80、443、51821；UDP51821端口出网 #可能需要出网的包括TCP53、8085、1883、8883、8083、18083; UDP53 文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.2 netmaker连通性验证

文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.3 防火墙设置

文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.4 k3s服务搭建

文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.4.0 Master

3.4.0为本地测试后加入的，可能和之前的IP不一致,请按需修改

文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.4.1 Master01

文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.4.2 Master02

文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.4.3 Master03

文 章 源 自 note.t4x.orgByrd's Blog-https://note.t4x.org/project/note-website-architecture/

3.4.4 worker节点

3.5 集群状态

3.6 calico配置

3.7 k3s状态

3.8 集群验证

四、服务配置

4.0 MySQL

4.1 MySQL Master

4.2 MySQL slave

4.1 MySQL验证

4.2 监控部署

4.3 网站配置

说明：此处通过nodeSelector选择了某节点模拟web的共享目录,通过nginx模拟了LB,没配置ingress。
通过访问http://jp.t4x.org:8080/安装了wordpress,可能存在网站无法写入 wp-config.php 的权限问题,可以通过配置网站目录权限解决。

$ cat nginx.yaml apiVersion: apps/v1 # 指定API版本 kind: Deployment # 资源的类型,Deployment StatefulSet Service metadata: name: blog # Deployment的名称 namespace: default # 指定namespace名称 spec: # Deployment 规格 replicas: 1 # 指定运行的Pod副本数 selector: # 选择器，用于确定哪些 Pod 受此 Deployment 管理 matchLabels: app: blog # 标签选择器，用于选择要控制的Pods, Pod 必须具有此标签才能被选择 environment: prod version: v1.0 role: internal template: # Pod 模板 metadata: labels: app: blog # Pod 的标签,与上面的选择器匹配 environment: prod role: internal version: v1.0 spec: nodeSelector: kubernetes.io/hostname: singapore-worker containers: - name: nginx # 容器的名称 image: bitnami/openresty:1.27.1-1 # 容器使用的镜像 imagePullPolicy: IfNotPresent ports: - containerPort: 80 # 容器监听的端口 - containerPort: 443 # 容器监听的端口 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: # 卷挂载 - mountPath: /opt/bitnami/openresty/nginx/conf/nginx.conf # 容器内的挂载路径 name: web-nginx-config # 引用的卷名称，必须与下面的卷名称匹配 - mountPath: /app # 容器内的挂载路径 name: web-files # 引用的卷名称，必须与下面的卷名称匹配 - name: php # 容器的名称 image: bitnami/php-fpm:7.4.33 # 容器使用的镜像 imagePullPolicy: IfNotPresent ports: - containerPort: 9000 # 容器监听的端口 resources: limits: memory: 200Mi requests: cpu: 200m memory: 200Mi volumeMounts: # 卷挂载 - mountPath: /app # 容器内的挂载路径 name: web-files # 引用的卷名称，必须与下面的卷名称匹配 volumes: # 卷列表 - name: web-nginx-config hostPath: # 使用 hostPath 卷类型 path: /etc/nginx/blog/conf/nginx.conf # 宿主机上的路径 type: File # 如果路径不存在，则创建它 #type: DirectoryOrCreate # 如果路径不存在，则创建它 - name: web-files # 卷的名称 hostPath: # 使用 hostPath 卷类型 path: /web/wordpress # 宿主机上的路径 type: DirectoryOrCreate # 如果路径不存在，则创建它 --- apiVersion: v1 # 指定了当前使用的Kubernetes API的版本 kind: Service # 定义了我们将创建一个Service metadata: # 包含了Service的元数据 name: blog-internal # 这是Service的名称 namespace: default #指定namespace名称 spec: #定义了Service的特性 type: NodePort # Service类型为NodePort selector: # 标签选择器，用于指定哪些Pod接收流量 app: blog # Pod 的标签,与上面的选择器匹配 environment: prod role: internal version: v1.0 ports: # 端口定义 - name: http protocol: TCP # 使用的协议 port: 8080 # Service对外暴露的端口 targetPort: 80 # Pod上的端口 - name: https protocol: TCP port: 443 targetPort: 443 - name: php protocol: TCP port: 9000 targetPort: 9000 $ kubectl create -f /root/k3s/nginx/nginx.yaml

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

$ cat nginx.yaml apiVersion: apps/v1 # 指定API版本 kind: Deployment # 资源的类型,Deployment StatefulSet Service metadata:   name: blog    # Deployment的名称   namespace: default    # 指定namespace名称 spec:    # Deployment 规格   replicas: 1 # 指定运行的Pod副本数   selector:    # 选择器，用于确定哪些 Pod 受此 Deployment 管理     matchLabels:       app: blog # 标签选择器，用于选择要控制的Pods, Pod 必须具有此标签才能被选择       environment: prod       version: v1.0       role: internal   template:    # Pod 模板     metadata:       labels:         app: blog # Pod 的标签,与上面的选择器匹配         environment: prod         role: internal         version: v1.0     spec:       nodeSelector:         kubernetes.io/hostname: singapore-worker       containers:       - name: nginx # 容器的名称         image: bitnami/openresty:1.27.1-1    # 容器使用的镜像         imagePullPolicy: IfNotPresent         ports:         - containerPort: 80 # 容器监听的端口         - containerPort: 443 # 容器监听的端口           resources:           limits:             memory: 200Mi           requests:             cpu: 100m             memory: 200Mi           volumeMounts:    # 卷挂载         - mountPath: /opt/bitnami/openresty/nginx/conf/nginx.conf  # 容器内的挂载路径           name: web-nginx-config  # 引用的卷名称，必须与下面的卷名称匹配         - mountPath: /app  # 容器内的挂载路径           name: web-files  # 引用的卷名称，必须与下面的卷名称匹配         - name: php # 容器的名称         image: bitnami/php-fpm:7.4.33    # 容器使用的镜像         imagePullPolicy: IfNotPresent         ports:         - containerPort: 9000 # 容器监听的端口         resources:           limits:             memory: 200Mi           requests:             cpu: 200m             memory: 200Mi         volumeMounts:    # 卷挂载         - mountPath: /app  # 容器内的挂载路径           name: web-files  # 引用的卷名称，必须与下面的卷名称匹配         volumes:    # 卷列表       - name: web-nginx-config         hostPath:  # 使用 hostPath 卷类型           path: /etc/nginx/blog/conf/nginx.conf  # 宿主机上的路径           type: File  # 如果路径不存在，则创建它           #type: DirectoryOrCreate  # 如果路径不存在，则创建它       - name: web-files  # 卷的名称         hostPath:  # 使用 hostPath 卷类型           path: /web/wordpress  # 宿主机上的路径           type: DirectoryOrCreate  # 如果路径不存在，则创建它 --- apiVersion: v1    # 指定了当前使用的Kubernetes API的版本 kind: Service    # 定义了我们将创建一个Service metadata:    # 包含了Service的元数据   name: blog-internal    # 这是Service的名称   namespace: default    #指定namespace名称 spec:    #定义了Service的特性   type: NodePort    # Service类型为NodePort   selector:         # 标签选择器，用于指定哪些Pod接收流量     app: blog # Pod 的标签,与上面的选择器匹配     environment: prod     role: internal     version: v1.0   ports:    # 端口定义     - name: http       protocol: TCP # 使用的协议       port: 8080    # Service对外暴露的端口       targetPort: 80    # Pod上的端口     - name: https       protocol: TCP       port: 443       targetPort: 443     - name: php       protocol: TCP       port: 9000       targetPort: 9000 $ kubectl create -f /root/k3s/nginx/nginx.yaml

nginx配置

$ cat /etc/nginx/blog/conf/nginx.conf #容器中nginx配置 events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root /app; index index.php; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } location ~ \.php { root /app; try_files $uri =404; fastcgi_pass blog-internal:9000; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi.conf; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } } $ cat /usr/local/openresty/nginx/conf/extra/pro.conf #模拟lb配置 server { listen 8080; server_name jp.t4x.org; location / { # 注意: 反向代理后端 URL 的最后需要有一个路径符号 client_max_body_size 500m; #proxy_pass http://100.64.0.4:42012/; proxy_pass http://100.64.0.4:42027; #42027是nodeport隐射的端口 proxy_set_header Host $host:8080; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } }

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

$ cat /etc/nginx/blog/conf/nginx.conf    #容器中nginx配置 events {     worker_connections  1024; } http {     include       mime.types;     default_type  application/octet-stream;     sendfile        on;     keepalive_timeout  65;     server {         listen       80;         server_name  localhost;         location / {             root   /app;             index  index.php;             proxy_set_header Host $host;             proxy_set_header X-Real-IP $remote_addr;             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;             proxy_set_header X-Forwarded-Proto $scheme;         }         location ~ \.php {             root           /app;             try_files $uri =404;             fastcgi_pass   blog-internal:9000;             fastcgi_index  index.php;             fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;             include        fastcgi.conf;         }         error_page   500 502 503 504  /50x.html;         location = /50x.html {             root   html;         }     } }   $ cat /usr/local/openresty/nginx/conf/extra/pro.conf    #模拟lb配置     server {         listen       8080;         server_name  jp.t4x.org;     location / {         # 注意: 反向代理后端 URL 的最后需要有一个路径符号         client_max_body_size 500m;         #proxy_pass http://100.64.0.4:42012/;         proxy_pass http://100.64.0.4:42027;    #42027是nodeport隐射的端口         proxy_set_header Host $host:8080;         proxy_set_header X-Real-IP $remote_addr;         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;         proxy_set_header X-Forwarded-Proto $scheme;     }     }

wordpress 数据库配置

$ kubectl exec -ti mysql-0 -- bash mysql> CREATE USER 'wordpress'@'%' IDENTIFIED BY 'admin'; mysql> CREATE DATABASE wordpress CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; mysql> GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'%' WITH GRANT OPTION; mysql> ALTER USER 'wordpress'@'%' IDENTIFIED WITH mysql_native_password BY 'admin'; /** Database username */ define( 'DB_USER', 'wordpress' ); /** Database password */ define( 'DB_PASSWORD', 'admin' ); /** Database hostname */ define( 'DB_HOST', 'mysql' ); /** Database charset to use in creating database tables. */ define( 'DB_CHARSET', 'utf8mb4' ); /** The database collate type. Don't change this if in doubt. */ define( 'DB_COLLATE', '' );

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

$ kubectl exec -ti mysql-0 -- bash mysql> CREATE USER 'wordpress'@'%' IDENTIFIED BY 'admin'; mysql> CREATE DATABASE wordpress CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; mysql> GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'%' WITH GRANT OPTION; mysql> ALTER USER 'wordpress'@'%' IDENTIFIED WITH mysql_native_password BY 'admin';   /** Database username */ define( 'DB_USER', 'wordpress' );   /** Database password */ define( 'DB_PASSWORD', 'admin' );   /** Database hostname */ define( 'DB_HOST', 'mysql' );   /** Database charset to use in creating database tables. */ define( 'DB_CHARSET', 'utf8mb4' );   /** The database collate type. Don't change this if in doubt. */ define( 'DB_COLLATE', '' );

五、服务器资源

root@jp-master01:~/k3s/nginx# free -m total used free shared buff/cache available Mem: 16002 3143 6261 3 6930 12858 Swap: 0 0 0 root@tw-master02:~# free -m total used free shared buff/cache available Mem: 16002 2260 9829 2 4243 13742 Swap: 0 0 0 root@hk-master03:~# free -m total used free shared buff/cache available Mem: 16002 2393 9589 1 4350 13608 Swap: 0 root@singapore-worker:/web/wordpress# free -m total used free shared buff/cache available Mem: 3924 972 2212 26 1035 2952 Swap: 0 0 0 root@jp-master01:~/k3s/nginx# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% hk-master03 88m 2% 2781Mi 17% jp-master01 213m 5% 4183Mi 26% singapore-worker 77m 3% 1103Mi 28% tw-master02 87m 2% 2737Mi 17%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

root@jp-master01:~/k3s/nginx# free -m                total        used        free      shared  buff/cache   available Mem:           16002        3143        6261           3        6930       12858 Swap:              0           0           0   root@tw-master02:~# free -m                total        used        free      shared  buff/cache   available Mem:           16002        2260        9829           2        4243       13742 Swap:              0           0           0   root@hk-master03:~# free -m                total        used        free      shared  buff/cache   available Mem:           16002        2393        9589           1        4350       13608 Swap:              0       root@singapore-worker:/web/wordpress# free -m                total        used        free      shared  buff/cache   available Mem:            3924         972        2212          26        1035        2952 Swap:              0           0           0   root@jp-master01:~/k3s/nginx# kubectl top node NAME               CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   hk-master03        88m          2%     2781Mi          17%       jp-master01        213m         5%     4183Mi          26%       singapore-worker   77m          3%     1103Mi          28%       tw-master02        87m          2%     2737Mi          17%

root@jp-master01:~/k3s/nginx# kubectl get nodes,pods,services,deployments,replicasets,statefulsets,daemonsets,jobs,cronjobs,configmaps,secrets,ingress,pv,pvc --all-namespaces NAME STATUS ROLES AGE VERSION node/hk-master03 Ready control-plane,etcd,master 4d13h v1.31.2+k3s1 node/jp-master01 Ready control-plane,etcd,master 5d3h v1.31.2+k3s1 node/singapore-worker Ready <none> 4d13h v1.31.2+k3s1 node/tw-master02 Ready control-plane,etcd,master 4d14h v1.31.2+k3s1 NAMESPACE NAME READY STATUS RESTARTS AGE default pod/blog-648664bcdd-2ttv9 2/2 Running 0 37m default pod/mysql-0 1/1 Running 0 21h default pod/slave-0 1/1 Running 2 (21h ago) 21h default pod/slave-1 1/1 Running 0 21h kube-system pod/calico-kube-controllers-5d7d9cdfd8-dwkzv 1/1 Running 0 4d1h kube-system pod/calico-node-2tvpc 1/1 Running 0 4d1h kube-system pod/calico-node-42wcv 1/1 Running 0 4d1h kube-system pod/calico-node-8wsd7 1/1 Running 0 4d1h kube-system pod/calico-node-smvd6 1/1 Running 0 4d1h kube-system pod/coredns-56f6fc8fd7-grnr4 1/1 Running 0 4d kube-system pod/local-path-provisioner-5cf85fd84d-wv24x 1/1 Running 0 4d1h kube-system pod/metrics-server-5985cbc9d7-jdc2w 1/1 Running 0 4d monitoring pod/alertmanager-main-0 2/2 Running 0 20h monitoring pod/alertmanager-main-1 2/2 Running 0 20h monitoring pod/alertmanager-main-2 2/2 Running 0 20h monitoring pod/blackbox-exporter-d7779b7d4-m4lvs 3/3 Running 0 20h monitoring pod/grafana-778555f685-x2nxc 1/1 Running 0 20h monitoring pod/kube-state-metrics-74f55cf6d9-xktd5 3/3 Running 0 20h monitoring pod/node-exporter-59lqd 2/2 Running 0 20h monitoring pod/node-exporter-f52ht 2/2 Running 0 20h monitoring pod/node-exporter-ht822 2/2 Running 0 20h monitoring pod/node-exporter-j2t5m 2/2 Running 0 20h monitoring pod/prometheus-adapter-784f566c54-54mj2 1/1 Running 0 20h monitoring pod/prometheus-adapter-784f566c54-dmftk 1/1 Running 0 20h monitoring pod/prometheus-k8s-0 2/2 Running 0 20h monitoring pod/prometheus-operator-6c55f986bc-gx7h2 2/2 Running 0 20h NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/blog-internal NodePort 10.43.4.237 <none> 8080:42027/TCP,443:30248/TCP,9000:34153/TCP 37m default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 5d3h default service/mysql ClusterIP None <none> 3306/TCP 21h default service/pxc-cluster ClusterIP 10.43.162.9 <none> 4567/TCP,4568/TCP,4444/TCP 21h kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 5d3h kube-system service/kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 20h kube-system service/metrics-server ClusterIP 10.43.146.92 <none> 443/TCP 5d3h monitoring service/alertmanager-main ClusterIP 10.43.146.202 <none> 9093/TCP,8080/TCP 20h monitoring service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 20h monitoring service/blackbox-exporter ClusterIP 10.43.219.155 <none> 9115/TCP,19115/TCP 20h monitoring service/grafana NodePort 10.43.31.46 <none> 3000:35790/TCP 20h monitoring service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 20h monitoring service/node-exporter ClusterIP None <none> 9100/TCP 20h monitoring service/prometheus-adapter ClusterIP 10.43.80.176 <none> 443/TCP 20h monitoring service/prometheus-k8s NodePort 10.43.251.45 <none> 9090:42012/TCP,8080:37098/TCP 20h monitoring service/prometheus-operated ClusterIP None <none> 9090/TCP 20h monitoring service/prometheus-operator ClusterIP None <none> 8443/TCP 20h NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default deployment.apps/blog 1/1 1 1 37m kube-system deployment.apps/calico-kube-controllers 1/1 1 1 4d1h kube-system deployment.apps/coredns 1/1 1 1 5d3h kube-system deployment.apps/local-path-provisioner 1/1 1 1 5d3h kube-system deployment.apps/metrics-server 1/1 1 1 5d3h monitoring deployment.apps/blackbox-exporter 1/1 1 1 20h monitoring deployment.apps/grafana 1/1 1 1 20h monitoring deployment.apps/kube-state-metrics 1/1 1 1 20h monitoring deployment.apps/prometheus-adapter 2/2 2 2 20h monitoring deployment.apps/prometheus-operator 1/1 1 1 20h NAMESPACE NAME DESIRED CURRENT READY AGE default replicaset.apps/blog-648664bcdd 1 1 1 37m kube-system replicaset.apps/calico-kube-controllers-5d7d9cdfd8 1 1 1 4d1h kube-system replicaset.apps/coredns-56f6fc8fd7 1 1 1 5d3h kube-system replicaset.apps/local-path-provisioner-5cf85fd84d 1 1 1 5d3h kube-system replicaset.apps/metrics-server-5985cbc9d7 1 1 1 5d3h monitoring replicaset.apps/blackbox-exporter-d7779b7d4 1 1 1 20h monitoring replicaset.apps/grafana-778555f685 1 1 1 20h monitoring replicaset.apps/kube-state-metrics-74f55cf6d9 1 1 1 20h monitoring replicaset.apps/prometheus-adapter-784f566c54 2 2 2 20h monitoring replicaset.apps/prometheus-operator-6c55f986bc 1 1 1 20h NAMESPACE NAME READY AGE default statefulset.apps/mysql 1/1 21h default statefulset.apps/slave 2/2 21h monitoring statefulset.apps/alertmanager-main 3/3 20h monitoring statefulset.apps/prometheus-k8s 1/1 20h NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system daemonset.apps/calico-node 4 4 4 4 4 kubernetes.io/os=linux 4d1h monitoring daemonset.apps/node-exporter 4 4 4 4 4 kubernetes.io/os=linux 20h NAMESPACE NAME DATA AGE default configmap/kube-root-ca.crt 1 5d3h kube-node-lease configmap/kube-root-ca.crt 1 5d3h kube-public configmap/kube-root-ca.crt 1 5d3h kube-system configmap/calico-config 4 4d1h kube-system configmap/cluster-dns 2 5d3h kube-system configmap/coredns 2 5d3h kube-system configmap/extension-apiserver-authentication 6 5d3h kube-system configmap/kube-apiserver-legacy-service-account-token-tracking 1 5d3h kube-system configmap/kube-root-ca.crt 1 5d3h kube-system configmap/local-path-config 4 5d3h monitoring configmap/adapter-config 1 20h monitoring configmap/blackbox-exporter-configuration 1 20h monitoring configmap/grafana-dashboard-alertmanager-overview 1 20h monitoring configmap/grafana-dashboard-apiserver 1 20h monitoring configmap/grafana-dashboard-cluster-total 1 20h monitoring configmap/grafana-dashboard-controller-manager 1 20h monitoring configmap/grafana-dashboard-grafana-overview 1 20h monitoring configmap/grafana-dashboard-k8s-resources-cluster 1 20h monitoring configmap/grafana-dashboard-k8s-resources-multicluster 1 20h monitoring configmap/grafana-dashboard-k8s-resources-namespace 1 20h monitoring configmap/grafana-dashboard-k8s-resources-node 1 20h monitoring configmap/grafana-dashboard-k8s-resources-pod 1 20h monitoring configmap/grafana-dashboard-k8s-resources-workload 1 20h monitoring configmap/grafana-dashboard-k8s-resources-workloads-namespace 1 20h monitoring configmap/grafana-dashboard-kubelet 1 20h monitoring configmap/grafana-dashboard-namespace-by-pod 1 20h monitoring configmap/grafana-dashboard-namespace-by-workload 1 20h monitoring configmap/grafana-dashboard-node-cluster-rsrc-use 1 20h monitoring configmap/grafana-dashboard-node-rsrc-use 1 20h monitoring configmap/grafana-dashboard-nodes 1 20h monitoring configmap/grafana-dashboard-nodes-aix 1 20h monitoring configmap/grafana-dashboard-nodes-darwin 1 20h monitoring configmap/grafana-dashboard-persistentvolumesusage 1 20h monitoring configmap/grafana-dashboard-pod-total 1 20h monitoring configmap/grafana-dashboard-prometheus 1 20h monitoring configmap/grafana-dashboard-prometheus-remote-write 1 20h monitoring configmap/grafana-dashboard-proxy 1 20h monitoring configmap/grafana-dashboard-<span class='keyword-be keyword-inner'><a href="https://note.t4x.org/kubernetes/k8s-scheduler-service/" title="Scheduler" target="_blank">Scheduler</a></span> 1 20h monitoring configmap/grafana-dashboard-workload-total 1 20h monitoring configmap/grafana-dashboards 1 20h monitoring configmap/kube-root-ca.crt 1 20h monitoring configmap/prometheus-k8s-rulefiles-0 8 20h NAMESPACE NAME TYPE DATA AGE default secret/mysql-secret Opaque 1 4d kube-system secret/hk-master03.node-password.k3s Opaque 1 4d13h kube-system secret/jp-master01.node-password.k3s Opaque 1 5d3h kube-system secret/k3s-serving kubernetes.io/tls 2 5d3h kube-system secret/singapore-worker.node-password.k3s Opaque 1 4d13h kube-system secret/tw-master02.node-password.k3s Opaque 1 4d14h monitoring secret/alertmanager-main Opaque 1 20h monitoring secret/alertmanager-main-generated Opaque 1 20h monitoring secret/alertmanager-main-tls-assets-0 Opaque 0 20h monitoring secret/alertmanager-main-web-config Opaque 1 20h monitoring secret/grafana-config Opaque 1 20h monitoring secret/grafana-datasources Opaque 1 20h monitoring secret/prometheus-k8s Opaque 1 20h monitoring secret/prometheus-k8s-thanos-prometheus-http-client-file Opaque 1 20h monitoring secret/prometheus-k8s-tls-assets-0 Opaque 0 20h monitoring secret/prometheus-k8s-web-config Opaque 1 20h NAMESPACE NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE default persistentvolume/pv-pxc-storage 1Gi RWO Retain Bound default/pxc-pvc tt-test <unset> 11m NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE default persistentvolumeclaim/pxc-pvc Bound pv-pxc-storage 1Gi RWO tt-test <unset> 11m

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153

root@jp-master01:~/k3s/nginx# kubectl get nodes,pods,services,deployments,replicasets,statefulsets,daemonsets,jobs,cronjobs,configmaps,secrets,ingress,pv,pvc --all-namespaces NAME                    STATUS   ROLES                       AGE     VERSION node/hk-master03        Ready    control-plane,etcd,master   4d13h   v1.31.2+k3s1 node/jp-master01        Ready    control-plane,etcd,master   5d3h    v1.31.2+k3s1 node/singapore-worker   Ready    <none>                      4d13h   v1.31.2+k3s1 node/tw-master02        Ready    control-plane,etcd,master   4d14h   v1.31.2+k3s1   NAMESPACE     NAME                                           READY   STATUS    RESTARTS      AGE default       pod/blog-648664bcdd-2ttv9                      2/2     Running   0             37m default       pod/mysql-0                                    1/1     Running   0             21h default       pod/slave-0                                    1/1     Running   2 (21h ago)   21h default       pod/slave-1                                    1/1     Running   0             21h kube-system   pod/calico-kube-controllers-5d7d9cdfd8-dwkzv   1/1     Running   0             4d1h kube-system   pod/calico-node-2tvpc                          1/1     Running   0             4d1h kube-system   pod/calico-node-42wcv                          1/1     Running   0             4d1h kube-system   pod/calico-node-8wsd7                          1/1     Running   0             4d1h kube-system   pod/calico-node-smvd6                          1/1     Running   0             4d1h kube-system   pod/coredns-56f6fc8fd7-grnr4                   1/1     Running   0             4d kube-system   pod/local-path-provisioner-5cf85fd84d-wv24x    1/1     Running   0             4d1h kube-system   pod/metrics-server-5985cbc9d7-jdc2w            1/1     Running   0             4d monitoring    pod/alertmanager-main-0                        2/2     Running   0             20h monitoring    pod/alertmanager-main-1                        2/2     Running   0             20h monitoring    pod/alertmanager-main-2                        2/2     Running   0             20h monitoring    pod/blackbox-exporter-d7779b7d4-m4lvs          3/3     Running   0             20h monitoring    pod/grafana-778555f685-x2nxc                   1/1     Running   0             20h monitoring    pod/kube-state-metrics-74f55cf6d9-xktd5        3/3     Running   0             20h monitoring    pod/node-exporter-59lqd                        2/2     Running   0             20h monitoring    pod/node-exporter-f52ht                        2/2     Running   0             20h monitoring    pod/node-exporter-ht822                        2/2     Running   0             20h monitoring    pod/node-exporter-j2t5m                        2/2     Running   0             20h monitoring    pod/prometheus-adapter-784f566c54-54mj2        1/1     Running   0             20h monitoring    pod/prometheus-adapter-784f566c54-dmftk        1/1     Running   0             20h monitoring    pod/prometheus-k8s-0                           2/2     Running   0             20h monitoring    pod/prometheus-operator-6c55f986bc-gx7h2       2/2     Running   0             20h   NAMESPACE     NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                       AGE default       service/blog-internal           NodePort    10.43.4.237     <none>        8080:42027/TCP,443:30248/TCP,9000:34153/TCP   37m default       service/kubernetes              ClusterIP   10.43.0.1       <none>        443/TCP                                       5d3h default       service/mysql                   ClusterIP   None            <none>        3306/TCP                                      21h default       service/pxc-cluster             ClusterIP   10.43.162.9     <none>        4567/TCP,4568/TCP,4444/TCP                    21h kube-system   service/kube-dns                ClusterIP   10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP                        5d3h kube-system   service/kubelet                 ClusterIP   None            <none>        10250/TCP,10255/TCP,4194/TCP                  20h kube-system   service/metrics-server          ClusterIP   10.43.146.92    <none>        443/TCP                                       5d3h monitoring    service/alertmanager-main       ClusterIP   10.43.146.202   <none>        9093/TCP,8080/TCP                             20h monitoring    service/alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP                    20h monitoring    service/blackbox-exporter       ClusterIP   10.43.219.155   <none>        9115/TCP,19115/TCP                            20h monitoring    service/grafana                 NodePort    10.43.31.46     <none>        3000:35790/TCP                                20h monitoring    service/kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP                             20h monitoring    service/node-exporter           ClusterIP   None            <none>        9100/TCP                                      20h monitoring    service/prometheus-adapter      ClusterIP   10.43.80.176    <none>        443/TCP                                       20h monitoring    service/prometheus-k8s          NodePort    10.43.251.45    <none>        9090:42012/TCP,8080:37098/TCP                 20h monitoring    service/prometheus-operated     ClusterIP   None            <none>        9090/TCP                                      20h monitoring    service/prometheus-operator     ClusterIP   None            <none>        8443/TCP                                      20h   NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE default       deployment.apps/blog                      1/1     1            1           37m kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           4d1h kube-system   deployment.apps/coredns                   1/1     1            1           5d3h kube-system   deployment.apps/local-path-provisioner    1/1     1            1           5d3h kube-system   deployment.apps/metrics-server            1/1     1            1           5d3h monitoring    deployment.apps/blackbox-exporter         1/1     1            1           20h monitoring    deployment.apps/grafana                   1/1     1            1           20h monitoring    deployment.apps/kube-state-metrics        1/1     1            1           20h monitoring    deployment.apps/prometheus-adapter        2/2     2            2           20h monitoring    deployment.apps/prometheus-operator       1/1     1            1           20h   NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE default       replicaset.apps/blog-648664bcdd                      1         1         1       37m kube-system   replicaset.apps/calico-kube-controllers-5d7d9cdfd8   1         1         1       4d1h kube-system   replicaset.apps/coredns-56f6fc8fd7                   1         1         1       5d3h kube-system   replicaset.apps/local-path-provisioner-5cf85fd84d    1         1         1       5d3h kube-system   replicaset.apps/metrics-server-5985cbc9d7            1         1         1       5d3h monitoring    replicaset.apps/blackbox-exporter-d7779b7d4          1         1         1       20h monitoring    replicaset.apps/grafana-778555f685                   1         1         1       20h monitoring    replicaset.apps/kube-state-metrics-74f55cf6d9        1         1         1       20h monitoring    replicaset.apps/prometheus-adapter-784f566c54        2         2         2       20h monitoring    replicaset.apps/prometheus-operator-6c55f986bc       1         1         1       20h   NAMESPACE    NAME                                 READY   AGE default      statefulset.apps/mysql               1/1     21h default      statefulset.apps/slave               2/2     21h monitoring   statefulset.apps/alertmanager-main   3/3     20h monitoring   statefulset.apps/prometheus-k8s      1/1     20h   NAMESPACE     NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE kube-system   daemonset.apps/calico-node     4         4         4       4            4           kubernetes.io/os=linux   4d1h monitoring    daemonset.apps/node-exporter   4         4         4       4            4           kubernetes.io/os=linux   20h   NAMESPACE         NAME                                                             DATA   AGE default           configmap/kube-root-ca.crt                                       1      5d3h kube-node-lease   configmap/kube-root-ca.crt                                       1      5d3h kube-public       configmap/kube-root-ca.crt                                       1      5d3h kube-system       configmap/calico-config                                          4      4d1h kube-system       configmap/cluster-dns                                            2      5d3h kube-system       configmap/coredns                                                2      5d3h kube-system       configmap/extension-apiserver-authentication                     6      5d3h kube-system       configmap/kube-apiserver-legacy-service-account-token-tracking   1      5d3h kube-system       configmap/kube-root-ca.crt                                       1      5d3h kube-system       configmap/local-path-config                                      4      5d3h monitoring        configmap/adapter-config                                         1      20h monitoring        configmap/blackbox-exporter-configuration                        1      20h monitoring        configmap/grafana-dashboard-alertmanager-overview                1      20h monitoring        configmap/grafana-dashboard-apiserver                            1      20h monitoring        configmap/grafana-dashboard-cluster-total                        1      20h monitoring        configmap/grafana-dashboard-controller-manager                   1      20h monitoring        configmap/grafana-dashboard-grafana-overview                     1      20h monitoring        configmap/grafana-dashboard-k8s-resources-cluster                1      20h monitoring        configmap/grafana-dashboard-k8s-resources-multicluster           1      20h monitoring        configmap/grafana-dashboard-k8s-resources-namespace              1      20h monitoring        configmap/grafana-dashboard-k8s-resources-node                   1      20h monitoring        configmap/grafana-dashboard-k8s-resources-pod                    1      20h monitoring        configmap/grafana-dashboard-k8s-resources-workload               1      20h monitoring        configmap/grafana-dashboard-k8s-resources-workloads-namespace    1      20h monitoring        configmap/grafana-dashboard-kubelet                              1      20h monitoring        configmap/grafana-dashboard-namespace-by-pod                     1      20h monitoring        configmap/grafana-dashboard-namespace-by-workload                1      20h monitoring        configmap/grafana-dashboard-node-cluster-rsrc-use                1      20h monitoring        configmap/grafana-dashboard-node-rsrc-use                        1      20h monitoring        configmap/grafana-dashboard-nodes                                1      20h monitoring        configmap/grafana-dashboard-nodes-aix                            1      20h monitoring        configmap/grafana-dashboard-nodes-darwin                         1      20h monitoring        configmap/grafana-dashboard-persistentvolumesusage               1      20h monitoring        configmap/grafana-dashboard-pod-total                            1      20h monitoring        configmap/grafana-dashboard-prometheus                           1      20h monitoring        configmap/grafana-dashboard-prometheus-remote-write              1      20h monitoring        configmap/grafana-dashboard-proxy                                1      20h monitoring        configmap/grafana-dashboard-scheduler                            1      20h monitoring        configmap/grafana-dashboard-workload-total                       1      20h monitoring        configmap/grafana-dashboards                                     1      20h monitoring        configmap/kube-root-ca.crt                                       1      20h monitoring        configmap/prometheus-k8s-rulefiles-0                             8      20h   NAMESPACE     NAME                                                       TYPE                DATA   AGE default       secret/mysql-secret                                        Opaque              1      4d kube-system   secret/hk-master03.node-password.k3s                       Opaque              1      4d13h kube-system   secret/jp-master01.node-password.k3s                       Opaque              1      5d3h kube-system   secret/k3s-serving                                         kubernetes.io/tls   2      5d3h kube-system   secret/singapore-worker.node-password.k3s                  Opaque              1      4d13h kube-system   secret/tw-master02.node-password.k3s                       Opaque              1      4d14h monitoring    secret/alertmanager-main                                   Opaque              1      20h monitoring    secret/alertmanager-main-generated                         Opaque              1      20h monitoring    secret/alertmanager-main-tls-assets-0                      Opaque              0      20h monitoring    secret/alertmanager-main-web-config                        Opaque              1      20h monitoring    secret/grafana-config                                      Opaque              1      20h monitoring    secret/grafana-datasources                                 Opaque              1      20h monitoring    secret/prometheus-k8s                                      Opaque              1      20h monitoring    secret/prometheus-k8s-thanos-prometheus-http-client-file   Opaque              1      20h monitoring    secret/prometheus-k8s-tls-assets-0                         Opaque              0      20h monitoring    secret/prometheus-k8s-web-config                           Opaque              1      20h NAMESPACE   NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE default     persistentvolume/pv-pxc-storage   1Gi        RWO            Retain           Bound    default/pxc-pvc   tt-test        <unset>                          11m   NAMESPACE   NAME                            STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE default     persistentvolumeclaim/pxc-pvc   Bound    pv-pxc-storage   1Gi        RWO            tt-test        <unset>                 11m

六、证书问题

证书脚本:

$ cat 1.sh #!/usr/bin/env bash # Example K3s CA certificate generation script. # # This script will generate files sufficient to bootstrap K3s cluster certificate # authorities. By default, the script will create the required files under # /var/lib/rancher/k3s/server/tls, where they will be found and used by K3s during initial # cluster startup. Note that these files MUST be present before K3s is started the first # time; certificate data SHOULD NOT be changed once the cluster has been initialized. # # The output path may be overridden with the DATA_DIR environment variable. # # This script will also auto-generate certificates and keys for both root and intermediate # certificate authorities if none are found. # If you have existing certs, you must place then in `DATA_DIR/server/tls`. # If you have only an existing root CA, provide: # root-ca.pem # root-ca.key # If you have an existing root and intermediate CA, provide: # root-ca.pem # intermediate-ca.pem # intermediate-ca.key set -e umask 027 TIMESTAMP=$(date +%s) PRODUCT="${PRODUCT:-k3s}" DATA_DIR="${DATA_DIR:-/data/${PRODUCT}}" if type -t openssl-3 &>/dev/null; then OPENSSL=openssl-3 else OPENSSL=openssl fi echo "Using $(type -p ${OPENSSL}): $(${OPENSSL} version)" if ! ${OPENSSL} ecparam -name prime256v1 -genkey -noout -out /dev/null &>/dev/null; then echo "openssl not found or missing Elliptic Curve (ecparam) support." exit 1 fi ${OPENSSL} version | grep -qF 'OpenSSL 3' && OPENSSL_GENRSA_FLAGS=-traditional mkdir -p "${DATA_DIR}/server/tls/etcd" cd "${DATA_DIR}/server/tls" # Set up temporary openssl configuration mkdir -p ".ca/certs" trap "rm -rf .ca" EXIT touch .ca/index openssl rand -hex 8 > .ca/serial cat >.ca/config <<'EOF' [ca] default_ca = ca_default [ca_default] dir = ./.ca database = $dir/index serial = $dir/serial new_certs_dir = $dir/certs default_md = sha256 policy = policy_anything [policy_anything] commonName = supplied [req] distinguished_name = req_distinguished_name [req_distinguished_name] [v3_ca] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always basicConstraints = critical, CA:true keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign EOF # Don't overwrite the service account issuer key; we pass the key into both the controller-manager # and the apiserver instead of passing a cert list into the apiserver, so there's no facility for # rotation and things will get very angry if all the SA keys are invalidated. if [[ -e service.key ]]; then echo "Generating additional Kubernetes service account issuer RSA key" OLD_SERVICE_KEY="$(cat service.key)" else echo "Generating Kubernetes service account issuer RSA key" fi ${OPENSSL} genrsa ${OPENSSL_GENRSA_FLAGS:-} -out service.key 2048 echo "${OLD_SERVICE_KEY}" >> service.key # Use existing root CA if present if [[ -e root-ca.pem ]]; then echo "Using existing root certificate" else echo "Generating root certificate authority RSA key and certificate" ${OPENSSL} genrsa ${OPENSSL_GENRSA_FLAGS:-} -out root-ca.key 4096 ${OPENSSL} req -x509 -new -nodes -sha256 -days 36500 \ -subj "/CN=${PRODUCT}-root-ca@${TIMESTAMP}" \ -key root-ca.key \ -out root-ca.pem \ -config .ca/config \ -extensions v3_ca fi cat root-ca.pem > root-ca.crt # Use existing intermediate CA if present if [[ -e intermediate-ca.pem ]]; then echo "Using existing intermediate certificate" else if [[ ! -e root-ca.key ]]; then echo "Cannot generate intermediate certificate without root certificate private key" exit 1 fi echo "Generating intermediate certificate authority RSA key and certificate" ${OPENSSL} genrsa ${OPENSSL_GENRSA_FLAGS:-} -out intermediate-ca.key 4096 ${OPENSSL} req -new -nodes \ -subj "/CN=${PRODUCT}-intermediate-ca@${TIMESTAMP}" \ -key intermediate-ca.key | ${OPENSSL} ca -batch -notext -days 36500 \ -in /dev/stdin \ -out intermediate-ca.pem \ -keyfile root-ca.key \ -cert root-ca.pem \ -config .ca/config \ -extensions v3_ca fi cat intermediate-ca.pem root-ca.pem > intermediate-ca.crt if [[ ! -e intermediate-ca.key ]]; then echo "Cannot generate leaf certificates without intermediate certificate private key" exit 1 fi # Generate new leaf CAs for all the control-plane and etcd components for TYPE in client server request-header etcd/peer etcd/server; do CERT_NAME="${PRODUCT}-$(echo ${TYPE} | tr / -)-ca" echo "Generating ${CERT_NAME} leaf certificate authority EC key and certificate" ${OPENSSL} ecparam -name prime256v1 -genkey -noout -out ${TYPE}-ca.key ${OPENSSL} req -new -nodes \ -subj "/CN=${CERT_NAME}@${TIMESTAMP}" \ -key ${TYPE}-ca.key | ${OPENSSL} ca -batch -notext -days 36500 \ -in /dev/stdin \ -out ${TYPE}-ca.pem \ -keyfile intermediate-ca.key \ -cert intermediate-ca.pem \ -config .ca/config \ -extensions v3_ca cat ${TYPE}-ca.pem \ intermediate-ca.pem \ root-ca.pem > ${TYPE}-ca.crt done echo echo "CA certificate generation complete. Required files are now present in: ${DATA_DIR}/server/tls" echo "For security purposes, you should make a secure copy of the following files and remove them from cluster members:" ls ${DATA_DIR}/server/tls/root-ca.* ${DATA_DIR}/server/tls/intermediate-ca.* | xargs -n1 echo -e "\t" if [ "${DATA_DIR}" != "/var/lib/rancher/${PRODUCT}" ]; then echo echo "To update certificates on an existing cluster, you may now run:" echo " k3s certificate rotate-ca --path=${DATA_DIR}/server" fi

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161

$ cat 1.sh #!/usr/bin/env bash   # Example K3s CA certificate generation script. # # This script will generate files sufficient to bootstrap K3s cluster certificate # authorities.  By default, the script will create the required files under # /var/lib/rancher/k3s/server/tls, where they will be found and used by K3s during initial # cluster startup. Note that these files MUST be present before K3s is started the first # time; certificate data SHOULD NOT be changed once the cluster has been initialized. # # The output path may be overridden with the DATA_DIR environment variable. # # This script will also auto-generate certificates and keys for both root and intermediate # certificate authorities if none are found. # If you have existing certs, you must place then in `DATA_DIR/server/tls`. # If you have only an existing root CA, provide: #   root-ca.pem #   root-ca.key # If you have an existing root and intermediate CA, provide: #   root-ca.pem #   intermediate-ca.pem #   intermediate-ca.key   set -e umask 027   TIMESTAMP=$(date +%s) PRODUCT="${PRODUCT:-k3s}" DATA_DIR="${DATA_DIR:-/data/${PRODUCT}}"   if type -t openssl-3 &>/dev/null; then   OPENSSL=openssl-3 else   OPENSSL=openssl fi   echo "Using $(type -p ${OPENSSL}): $(${OPENSSL} version)"   if ! ${OPENSSL} ecparam -name prime256v1 -genkey -noout -out /dev/null &>/dev/null; then   echo "openssl not found or missing Elliptic Curve (ecparam) support."   exit 1 fi   ${OPENSSL} version | grep -qF 'OpenSSL 3' && OPENSSL_GENRSA_FLAGS=-traditional   mkdir -p "${DATA_DIR}/server/tls/etcd" cd "${DATA_DIR}/server/tls"   # Set up temporary openssl configuration mkdir -p ".ca/certs" trap "rm -rf .ca" EXIT touch .ca/index openssl rand -hex 8 > .ca/serial cat >.ca/config <<'EOF' [ca] default_ca = ca_default [ca_default] dir = ./.ca database = $dir/index serial = $dir/serial new_certs_dir = $dir/certs default_md = sha256 policy = policy_anything [policy_anything] commonName = supplied [req] distinguished_name = req_distinguished_name [req_distinguished_name] [v3_ca] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always basicConstraints = critical, CA:true keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign EOF   # Don't overwrite the service account issuer key; we pass the key into both the controller-manager # and the apiserver instead of passing a cert list into the apiserver, so there's no facility for # rotation and things will get very angry if all the SA keys are invalidated. if [[ -e service.key ]]; then   echo "Generating additional Kubernetes service account issuer RSA key"   OLD_SERVICE_KEY="$(cat service.key)" else   echo "Generating Kubernetes service account issuer RSA key" fi ${OPENSSL} genrsa ${OPENSSL_GENRSA_FLAGS:-} -out service.key 2048 echo "${OLD_SERVICE_KEY}" >> service.key   # Use existing root CA if present if [[ -e root-ca.pem ]]; then   echo "Using existing root certificate" else   echo "Generating root certificate authority RSA key and certificate"   ${OPENSSL} genrsa ${OPENSSL_GENRSA_FLAGS:-} -out root-ca.key 4096   ${OPENSSL} req -x509 -new -nodes -sha256 -days 36500 \                  -subj "/CN=${PRODUCT}-root-ca@${TIMESTAMP}" \                  -key root-ca.key \                  -out root-ca.pem \                  -config .ca/config \                  -extensions v3_ca fi cat root-ca.pem > root-ca.crt   # Use existing intermediate CA if present if [[ -e intermediate-ca.pem ]]; then   echo "Using existing intermediate certificate" else   if [[ ! -e root-ca.key ]]; then     echo "Cannot generate intermediate certificate without root certificate private key"     exit 1   fi     echo "Generating intermediate certificate authority RSA key and certificate"   ${OPENSSL} genrsa ${OPENSSL_GENRSA_FLAGS:-} -out intermediate-ca.key 4096   ${OPENSSL} req -new -nodes \                  -subj "/CN=${PRODUCT}-intermediate-ca@${TIMESTAMP}" \                  -key intermediate-ca.key |   ${OPENSSL} ca  -batch -notext -days 36500 \                  -in /dev/stdin \                  -out intermediate-ca.pem \                  -keyfile root-ca.key \                  -cert root-ca.pem \                  -config .ca/config \                  -extensions v3_ca fi cat intermediate-ca.pem root-ca.pem > intermediate-ca.crt   if [[ ! -e intermediate-ca.key ]]; then   echo "Cannot generate leaf certificates without intermediate certificate private key"   exit 1 fi   # Generate new leaf CAs for all the control-plane and etcd components for TYPE in client server request-header etcd/peer etcd/server; do   CERT_NAME="${PRODUCT}-$(echo ${TYPE} | tr / -)-ca"   echo "Generating ${CERT_NAME} leaf certificate authority EC key and certificate"   ${OPENSSL} ecparam -name prime256v1 -genkey -noout -out ${TYPE}-ca.key   ${OPENSSL} req -new -nodes \                  -subj "/CN=${CERT_NAME}@${TIMESTAMP}" \                  -key ${TYPE}-ca.key |   ${OPENSSL} ca  -batch -notext -days 36500 \                  -in /dev/stdin \                  -out ${TYPE}-ca.pem \                  -keyfile intermediate-ca.key \                  -cert intermediate-ca.pem \                  -config .ca/config \                  -extensions v3_ca   cat ${TYPE}-ca.pem \       intermediate-ca.pem \       root-ca.pem > ${TYPE}-ca.crt done   echo echo "CA certificate generation complete. Required files are now present in: ${DATA_DIR}/server/tls" echo "For security purposes, you should make a secure copy of the following files and remove them from cluster members:" ls ${DATA_DIR}/server/tls/root-ca.* ${DATA_DIR}/server/tls/intermediate-ca.* | xargs -n1 echo -e "\t"   if [ "${DATA_DIR}" != "/var/lib/rancher/${PRODUCT}" ]; then   echo   echo "To update certificates on an existing cluster, you may now run:"   echo "    k3s certificate rotate-ca --path=${DATA_DIR}/server" fi

有效期：

$ for i in `find /data/k3s/server/tls/ -name "*.crt"`; do echo $i; openssl x509 -enddate -noout -in $i; done /data/k3s/server/tls/etcd/peer-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/etcd/server-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/etcd/client.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/etcd/peer-server-client.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/etcd/server-client.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/root-ca.crt notAfter=Dec 24 14:11:09 2124 GMT /data/k3s/server/tls/intermediate-ca.crt notAfter=Dec 24 14:11:11 2124 GMT /data/k3s/server/tls/client-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/server-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/request-header-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/client-ca.nochain.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/client-admin.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-supervisor.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-controller.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-scheduler.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-kube-apiserver.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-kube-proxy.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-k3s-controller.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-k3s-cloud-controller.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/server-ca.nochain.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/serving-kube-apiserver.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-auth-proxy.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/temporary-certs/apiserver-loopback-client__.crt notAfter=Dec 24 13:19:44 2124 GMT

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

$ for i in `find /data/k3s/server/tls/ -name "*.crt"`; do echo $i; openssl x509 -enddate -noout -in $i; done /data/k3s/server/tls/etcd/peer-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/etcd/server-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/etcd/client.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/etcd/peer-server-client.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/etcd/server-client.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/root-ca.crt notAfter=Dec 24 14:11:09 2124 GMT /data/k3s/server/tls/intermediate-ca.crt notAfter=Dec 24 14:11:11 2124 GMT /data/k3s/server/tls/client-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/server-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/request-header-ca.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/client-ca.nochain.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/client-admin.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-supervisor.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-controller.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-scheduler.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-kube-apiserver.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-kube-proxy.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-k3s-controller.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-k3s-cloud-controller.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/server-ca.nochain.crt notAfter=Dec 24 14:11:12 2124 GMT /data/k3s/server/tls/serving-kube-apiserver.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/client-auth-proxy.crt notAfter=Dec 24 14:19:43 2124 GMT /data/k3s/server/tls/temporary-certs/apiserver-loopback-client__.crt notAfter=Dec 24 13:19:44 2124 GMT

证书：

MasterNodes='k3s-master02 k3s-master03' for NODE in $MasterNodes; do ssh $NODE "mkdir -p /etc/etcd/ssl" for FILE in *; do scp -rp /data/k3s/server/tls/${FILE} $NODE:/data/k3s/server/tls/${FILE} done done

0 1 2 3 4 5 6

MasterNodes='k3s-master02 k3s-master03' for NODE in $MasterNodes; do ssh $NODE "mkdir -p /etc/etcd/ssl" for FILE in *; do    scp -rp /data/k3s/server/tls/${FILE} $NODE:/data/k3s/server/tls/${FILE} done done

七、local-path

$ kubectl get storageclass -n kube-system #默认是delete策略,保留数据更改为Retain策略 NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 25h $ kubectl get sc -A NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-path (default) rancher.io/local-path Retain WaitForFirstConsumer false 58s $ kubectl delete -f mysql-storage.yaml $ cat mysql-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pxc-data namespace: db spec: accessModes: - ReadWriteOnce storageClassName: local-path resources: requests: storage: 1Gi $ kubectl create -f mysql-pvc.yaml $ kubectl get pv,pvc -A #pending状态,$ kubectl describe pvc localpath-pxc -ndb 提示 "waiting for first consumer to be created before binding" NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE db persistentvolumeclaim/pxc-data Pending $ cat mysql-master1.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql namespace: db spec: serviceName: mysql replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: nodeSelector: kubernetes.io/hostname: k3s-master01 volumes: - name: pxc-storange persistentVolumeClaim: claimName: pxc-data containers: - name: mysql image: percona/percona-xtradb-cluster:5.7.44 #image: bitnami/openresty:1.27.1-1 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: MYSQL_ROOT_PASSWORD - name: CLUSTER_NAME value: "PXC" ports: - containerPort: 3306 - containerPort: 4444 - containerPort: 4567 - containerPort: 4568 volumeMounts: - name: pxc-storange mountPath: /var/lib/mysql tolerations: - effect: NoExecute key: node.kubernetes.io/unreachable tolerationSeconds: 10 operator: Exists - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 10 - key: nodetype operator: Equal value: "master" affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - mysql namespaces: - db topologyKey: kubernetes.io/hostname --- apiVersion: v1 kind: Service metadata: name: mysql namespace: db spec: type: ClusterIP #clusterIP: None # Headless Service selector: app: mysql ports: - port: 3306 targetPort: 3306 --- apiVersion: v1 kind: Service metadata: name: pxc-cluster namespace: db spec: type: ClusterIP selector: app: mysql ports: - name: pxc-0 protocol: TCP port: 4567 targetPort: 4567 - name: pxc-1 protocol: TCP port: 4568 targetPort: 4568 - name: pxc-2 protocol: TCP port: 4444 targetPort: 4444 $ kubectl get pv,pvc -A NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-f522f581-b575-4568-a9ff-1ba85d9c4fa2 1Gi RWO Retain Bound db/pxc-data local-path <unset> 93s NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE db persistentvolumeclaim/pxc-data Bound pvc-f522f581-b575-4568-a9ff-1ba85d9c4fa2 1Gi RWO local-path <unset> 3m49s $ kubectl get pod -ndb NAME READY STATUS RESTARTS AGE mysql-0 1/1 Running 0 3m19s $ kubectl exec -ti mysql-0 -ndb -- mysql -uroot -p'admin123' mysql>

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138

$ kubectl get storageclass -n kube-system    #默认是delete策略,保留数据更改为Retain策略 NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  25h $ kubectl get sc -A NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE local-path (default)   rancher.io/local-path   Retain          WaitForFirstConsumer   false                  58s $ kubectl delete -f mysql-storage.yaml $ cat mysql-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata:   name: pxc-data   namespace: db spec:   accessModes:     - ReadWriteOnce   storageClassName: local-path   resources:     requests:       storage: 1Gi $ kubectl create -f mysql-pvc.yaml $ kubectl get pv,pvc -A    #pending状态,$ kubectl describe pvc localpath-pxc -ndb 提示 "waiting for first consumer to be created before binding" NAMESPACE   NAME                             STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE db          persistentvolumeclaim/pxc-data   Pending $ cat mysql-master1.yaml apiVersion: apps/v1 kind: StatefulSet metadata:   name: mysql   namespace: db spec:   serviceName: mysql   replicas: 1   selector:     matchLabels:       app: mysql   template:     metadata:       labels:         app: mysql     spec:       nodeSelector:         kubernetes.io/hostname: k3s-master01       volumes:       - name: pxc-storange         persistentVolumeClaim:           claimName: pxc-data       containers:         - name: mysql           image: percona/percona-xtradb-cluster:5.7.44           #image: bitnami/openresty:1.27.1-1           env:             - name: MYSQL_ROOT_PASSWORD               valueFrom:                 secretKeyRef:                   name: mysql-secret                   key: MYSQL_ROOT_PASSWORD             - name: CLUSTER_NAME               value: "PXC"           ports:             - containerPort: 3306             - containerPort: 4444             - containerPort: 4567             - containerPort: 4568           volumeMounts:           - name: pxc-storange             mountPath: /var/lib/mysql       tolerations:       - effect: NoExecute         key: node.kubernetes.io/unreachable         tolerationSeconds: 10         operator: Exists       - effect: NoExecute         key: node.kubernetes.io/not-ready         operator: Exists         tolerationSeconds: 10       - key: nodetype         operator: Equal         value: "master"       affinity:         podAntiAffinity:           requiredDuringSchedulingIgnoredDuringExecution:           - labelSelector:               matchExpressions:               - key: app                 operator: In                 values:                 - mysql             namespaces:             - db             topologyKey: kubernetes.io/hostname --- apiVersion: v1 kind: Service metadata:   name: mysql   namespace: db spec:   type: ClusterIP   #clusterIP: None  # Headless Service   selector:     app: mysql   ports:   - port: 3306     targetPort: 3306 --- apiVersion: v1 kind: Service metadata:   name: pxc-cluster   namespace: db spec:   type: ClusterIP   selector:     app: mysql   ports:     - name: pxc-0       protocol: TCP       port: 4567       targetPort: 4567     - name: pxc-1       protocol: TCP       port: 4568       targetPort: 4568     - name: pxc-2       protocol: TCP       port: 4444       targetPort: 4444 $ kubectl get pv,pvc -A   NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM         STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE persistentvolume/pvc-f522f581-b575-4568-a9ff-1ba85d9c4fa2   1Gi        RWO            Retain           Bound    db/pxc-data   local-path     <unset>                          93s   NAMESPACE   NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE db          persistentvolumeclaim/pxc-data   Bound    pvc-f522f581-b575-4568-a9ff-1ba85d9c4fa2   1Gi        RWO            local-path     <unset>                 3m49s $ kubectl get pod -ndb NAME      READY   STATUS    RESTARTS   AGE mysql-0   1/1     Running   0          3m19s $ kubectl exec -ti mysql-0 -ndb -- mysql -uroot -p'admin123' mysql>

八、network

$ kubectl get nodes,pods,services,deployments,replicasets,statefulsets,daemonsets,jobs,cronjobs,configmaps,secrets,ingress,networkpolicy,storageclass,pv,pvc --all-namespaces #基于--flannel-backend 'vxlan' NAME STATUS ROLES AGE VERSION node/k3s-master01 Ready control-plane,etcd,master 32h v1.31.2+k3s1 node/k3s-master02 Ready control-plane,etcd,master 32h v1.31.2+k3s1 node/k3s-master03 Ready control-plane,etcd,master 32h v1.31.2+k3s1 node/k3s-node01 Ready worker 32h v1.31.2+k3s1 node/k3s-node02 Ready worker 32h v1.31.2+k3s1 NAMESPACE NAME READY STATUS RESTARTS AGE db pod/mysql-0 1/1 Running 1 (9m55s ago) 172m kube-system pod/coredns-56f6fc8fd7-9pqh9 1/1 Running 4 (9m55s ago) 32h kube-system pod/local-path-provisioner-5cf85fd84d-2dfg4 1/1 Running 1 (9m13s ago) 3h57m kube-system pod/metrics-server-5985cbc9d7-wwhb2 1/1 Running 4 (9m55s ago) 32h monitoring pod/alertmanager-main-0 2/2 Running 0 4m27s monitoring pod/alertmanager-main-1 2/2 Running 0 4m26s monitoring pod/alertmanager-main-2 2/2 Running 0 4m26s monitoring pod/blackbox-exporter-d7779b7d4-rjdk2 3/3 Running 0 7m32s monitoring pod/grafana-778555f685-gmkrz 1/1 Running 0 7m3s monitoring pod/kube-state-metrics-74f55cf6d9-mqpcg 3/3 Running 0 6m53s monitoring pod/node-exporter-2gx8h 2/2 Running 0 6m36s monitoring pod/node-exporter-hdsnn 2/2 Running 0 6m36s monitoring pod/node-exporter-htjtd 2/2 Running 0 6m35s monitoring pod/node-exporter-m8tpv 2/2 Running 0 6m36s monitoring pod/node-exporter-qxkhw 2/2 Running 0 6m35s monitoring pod/prometheus-adapter-784f566c54-dcdf5 1/1 Running 0 5m29s monitoring pod/prometheus-adapter-784f566c54-z7cz2 1/1 Running 0 5m29s monitoring pod/prometheus-k8s-0 2/2 Running 0 4m27s monitoring pod/prometheus-k8s-1 2/2 Running 0 4m26s monitoring pod/prometheus-operator-6c55f986bc-cfqd8 2/2 Running 0 5m16s NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE db service/mysql ClusterIP 10.243.187.137 <none> 3306/TCP,4567/TCP,4568/TCP,4444/TCP 172m default service/kubernetes ClusterIP 10.243.0.1 <none> 443/TCP 32h kube-system service/kube-dns ClusterIP 10.243.0.10 <none> 53/UDP,53/TCP,9153/TCP 32h kube-system service/kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 30h kube-system service/metrics-server ClusterIP 10.243.170.109 <none> 443/TCP 32h monitoring service/alertmanager-main ClusterIP 10.243.2.40 <none> 9093/TCP,8080/TCP 7m38s monitoring service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 4m29s monitoring service/blackbox-exporter ClusterIP 10.243.110.71 <none> 9115/TCP,19115/TCP 7m33s monitoring service/grafana NodePort 10.243.56.184 <none> 3000:37722/TCP 7m4s monitoring service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 6m54s monitoring service/node-exporter ClusterIP None <none> 9100/TCP 6m38s monitoring service/prometheus-adapter ClusterIP 10.243.188.73 <none> 443/TCP 5m36s monitoring service/prometheus-k8s NodePort 10.243.127.141 <none> 9090:35230/TCP,8080:31893/TCP 5m58s monitoring service/prometheus-operated ClusterIP None <none> 9090/TCP 4m28s monitoring service/prometheus-operator ClusterIP None <none> 8443/TCP 5m20s NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system deployment.apps/coredns 1/1 1 1 32h kube-system deployment.apps/local-path-provisioner 1/1 1 1 3h57m kube-system deployment.apps/metrics-server 1/1 1 1 32h monitoring deployment.apps/blackbox-exporter 1/1 1 1 7m36s monitoring deployment.apps/grafana 1/1 1 1 7m9s monitoring deployment.apps/kube-state-metrics 1/1 1 1 6m59s monitoring deployment.apps/prometheus-adapter 2/2 2 2 5m45s monitoring deployment.apps/prometheus-operator 1/1 1 1 5m28s NAMESPACE NAME DESIRED CURRENT READY AGE kube-system replicaset.apps/coredns-56f6fc8fd7 1 1 1 32h kube-system replicaset.apps/local-path-provisioner-5cf85fd84d 1 1 1 3h57m kube-system replicaset.apps/metrics-server-5985cbc9d7 1 1 1 32h monitoring replicaset.apps/blackbox-exporter-d7779b7d4 1 1 1 7m36s monitoring replicaset.apps/grafana-778555f685 1 1 1 7m8s monitoring replicaset.apps/kube-state-metrics-74f55cf6d9 1 1 1 6m59s monitoring replicaset.apps/prometheus-adapter-784f566c54 2 2 2 5m44s monitoring replicaset.apps/prometheus-operator-6c55f986bc 1 1 1 5m27s NAMESPACE NAME READY AGE db statefulset.apps/mysql 1/1 172m monitoring statefulset.apps/alertmanager-main 3/3 4m30s monitoring statefulset.apps/prometheus-k8s 2/2 4m29s NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE monitoring daemonset.apps/node-exporter 5 5 5 5 5 kubernetes.io/os=linux 6m44s NAMESPACE NAME DATA AGE db configmap/kube-root-ca.crt 1 28h default configmap/kube-root-ca.crt 1 32h kube-node-lease configmap/kube-root-ca.crt 1 32h kube-public configmap/kube-root-ca.crt 1 32h kube-system configmap/cluster-dns 2 32h kube-system configmap/coredns 2 32h kube-system configmap/extension-apiserver-authentication 6 32h kube-system configmap/kube-apiserver-legacy-service-account-token-tracking 1 32h kube-system configmap/kube-root-ca.crt 1 32h kube-system configmap/local-path-config 4 3h57m monitoring configmap/adapter-config 1 5m46s monitoring configmap/blackbox-exporter-configuration 1 7m36s monitoring configmap/grafana-dashboard-alertmanager-overview 1 7m31s monitoring configmap/grafana-dashboard-apiserver 1 7m30s monitoring configmap/grafana-dashboard-cluster-total 1 7m30s monitoring configmap/grafana-dashboard-controller-manager 1 7m29s monitoring configmap/grafana-dashboard-grafana-overview 1 7m28s monitoring configmap/grafana-dashboard-k8s-resources-cluster 1 7m27s monitoring configmap/grafana-dashboard-k8s-resources-multicluster 1 7m26s monitoring configmap/grafana-dashboard-k8s-resources-namespace 1 7m26s monitoring configmap/grafana-dashboard-k8s-resources-node 1 7m25s monitoring configmap/grafana-dashboard-k8s-resources-pod 1 7m25s monitoring configmap/grafana-dashboard-k8s-resources-workload 1 7m24s monitoring configmap/grafana-dashboard-k8s-resources-workloads-namespace 1 7m23s monitoring configmap/grafana-dashboard-kubelet 1 7m23s monitoring configmap/grafana-dashboard-namespace-by-pod 1 7m22s monitoring configmap/grafana-dashboard-namespace-by-workload 1 7m21s monitoring configmap/grafana-dashboard-node-cluster-rsrc-use 1 7m21s monitoring configmap/grafana-dashboard-node-rsrc-use 1 7m20s monitoring configmap/grafana-dashboard-nodes 1 7m17s monitoring configmap/grafana-dashboard-nodes-aix 1 7m19s monitoring configmap/grafana-dashboard-nodes-darwin 1 7m18s monitoring configmap/grafana-dashboard-persistentvolumesusage 1 7m16s monitoring configmap/grafana-dashboard-pod-total 1 7m15s monitoring configmap/grafana-dashboard-prometheus 1 7m14s monitoring configmap/grafana-dashboard-prometheus-remote-write 1 7m15s monitoring configmap/grafana-dashboard-proxy 1 7m12s monitoring configmap/grafana-dashboard-scheduler 1 7m11s monitoring configmap/grafana-dashboard-workload-total 1 7m11s monitoring configmap/grafana-dashboards 1 7m10s monitoring configmap/kube-root-ca.crt 1 8m13s monitoring configmap/prometheus-k8s-rulefiles-0 8 4m35s NAMESPACE NAME TYPE DATA AGE db secret/mysql-secret Opaque 1 26h kube-system secret/k3s-master01.node-password.k3s Opaque 1 32h kube-system secret/k3s-master02.node-password.k3s Opaque 1 32h kube-system secret/k3s-master03.node-password.k3s Opaque 1 32h kube-system secret/k3s-node01.node-password.k3s Opaque 1 32h kube-system secret/k3s-node02.node-password.k3s Opaque 1 32h kube-system secret/k3s-serving kubernetes.io/tls 2 32h monitoring secret/alertmanager-main Opaque 1 7m39s monitoring secret/alertmanager-main-generated Opaque 1 4m35s monitoring secret/alertmanager-main-tls-assets-0 Opaque 0 4m33s monitoring secret/alertmanager-main-web-config Opaque 1 4m32s monitoring secret/grafana-config Opaque 1 7m31s monitoring secret/grafana-datasources Opaque 1 7m31s monitoring secret/prometheus-k8s Opaque 1 4m33s monitoring secret/prometheus-k8s-thanos-prometheus-http-client-file Opaque 1 4m30s monitoring secret/prometheus-k8s-tls-assets-0 Opaque 0 4m32s monitoring secret/prometheus-k8s-web-config Opaque 1 4m30s NAMESPACE NAME POD-SELECTOR AGE monitoring networkpolicy.networking.k8s.io/alertmanager-main app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus 7m42s monitoring networkpolicy.networking.k8s.io/blackbox-exporter app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus 7m36s monitoring networkpolicy.networking.k8s.io/grafana app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus 7m9s monitoring networkpolicy.networking.k8s.io/kube-state-metrics app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus 6m59s monitoring networkpolicy.networking.k8s.io/node-exporter app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus 6m44s monitoring networkpolicy.networking.k8s.io/prometheus-adapter app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus 5m43s monitoring networkpolicy.networking.k8s.io/prometheus-k8s app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus 6m34s monitoring networkpolicy.networking.k8s.io/prometheus-operator app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus 5m26s NAMESPACE NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE storageclass.storage.k8s.io/local-path (default) rancher.io/local-path Retain WaitForFirstConsumer false 3h57m NAMESPACE NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-4a232f33-602b-490f-a14d-4fa894ed6bda 1Gi RWO Retain Bound db/pxc-data local-path <unset> 177m persistentvolume/pvc-c9c60b30-06f5-4eb8-b0a8-73539a358f7c 1Gi RWO Retain Released db/pxc-data local-path <unset> 3h22m persistentvolume/pvc-f522f581-b575-4568-a9ff-1ba85d9c4fa2 1Gi RWO Retain Released db/pxc-data local-path <unset> 3h43m NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE db persistentvolumeclaim/pxc-data Bound pvc-4a232f33-602b-490f-a14d-4fa894ed6bda 1Gi RWO local-path <unset> 179m $ cat /etc/cni/net.d/10-calico.conflist { "name": "k8s-pod-network", "cniVersion": "0.3.1", "plugins": [ { "type": "calico", "log_level": "info", "log_file_path": "/var/log/calico/cni/cni.log", "datastore_type": "kubernetes", "nodename": "k3s-master01", "mtu": 0, "ipam": { "type": "calico-ipam", "container_setings": {"allow_ip_frowarding": true} }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "portmap", "snat": true, "capabilities": {"portMappings": true} }, { "type": "bandwidth", "capabilities": {"bandwidth": true} } ] }

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194

$ kubectl get nodes,pods,services,deployments,replicasets,statefulsets,daemonsets,jobs,cronjobs,configmaps,secrets,ingress,networkpolicy,storageclass,pv,pvc --all-namespaces    #基于--flannel-backend 'vxlan' NAME                STATUS   ROLES                       AGE   VERSION node/k3s-master01   Ready    control-plane,etcd,master   32h   v1.31.2+k3s1 node/k3s-master02   Ready    control-plane,etcd,master   32h   v1.31.2+k3s1 node/k3s-master03   Ready    control-plane,etcd,master   32h   v1.31.2+k3s1 node/k3s-node01     Ready    worker                      32h   v1.31.2+k3s1 node/k3s-node02     Ready    worker                      32h   v1.31.2+k3s1   NAMESPACE     NAME                                          READY   STATUS    RESTARTS        AGE db            pod/mysql-0                                   1/1     Running   1 (9m55s ago)   172m kube-system   pod/coredns-56f6fc8fd7-9pqh9                  1/1     Running   4 (9m55s ago)   32h kube-system   pod/local-path-provisioner-5cf85fd84d-2dfg4   1/1     Running   1 (9m13s ago)   3h57m kube-system   pod/metrics-server-5985cbc9d7-wwhb2           1/1     Running   4 (9m55s ago)   32h monitoring    pod/alertmanager-main-0                       2/2     Running   0               4m27s monitoring    pod/alertmanager-main-1                       2/2     Running   0               4m26s monitoring    pod/alertmanager-main-2                       2/2     Running   0               4m26s monitoring    pod/blackbox-exporter-d7779b7d4-rjdk2         3/3     Running   0               7m32s monitoring    pod/grafana-778555f685-gmkrz                  1/1     Running   0               7m3s monitoring    pod/kube-state-metrics-74f55cf6d9-mqpcg       3/3     Running   0               6m53s monitoring    pod/node-exporter-2gx8h                       2/2     Running   0               6m36s monitoring    pod/node-exporter-hdsnn                       2/2     Running   0               6m36s monitoring    pod/node-exporter-htjtd                       2/2     Running   0               6m35s monitoring    pod/node-exporter-m8tpv                       2/2     Running   0               6m36s monitoring    pod/node-exporter-qxkhw                       2/2     Running   0               6m35s monitoring    pod/prometheus-adapter-784f566c54-dcdf5       1/1     Running   0               5m29s monitoring    pod/prometheus-adapter-784f566c54-z7cz2       1/1     Running   0               5m29s monitoring    pod/prometheus-k8s-0                          2/2     Running   0               4m27s monitoring    pod/prometheus-k8s-1                          2/2     Running   0               4m26s monitoring    pod/prometheus-operator-6c55f986bc-cfqd8      2/2     Running   0               5m16s   NAMESPACE     NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE db            service/mysql                   ClusterIP   10.243.187.137   <none>        3306/TCP,4567/TCP,4568/TCP,4444/TCP   172m default       service/kubernetes              ClusterIP   10.243.0.1       <none>        443/TCP                               32h kube-system   service/kube-dns                ClusterIP   10.243.0.10      <none>        53/UDP,53/TCP,9153/TCP                32h kube-system   service/kubelet                 ClusterIP   None             <none>        10250/TCP,10255/TCP,4194/TCP          30h kube-system   service/metrics-server          ClusterIP   10.243.170.109   <none>        443/TCP                               32h monitoring    service/alertmanager-main       ClusterIP   10.243.2.40      <none>        9093/TCP,8080/TCP                     7m38s monitoring    service/alertmanager-operated   ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP            4m29s monitoring    service/blackbox-exporter       ClusterIP   10.243.110.71    <none>        9115/TCP,19115/TCP                    7m33s monitoring    service/grafana                 NodePort    10.243.56.184    <none>        3000:37722/TCP                        7m4s monitoring    service/kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP                     6m54s monitoring    service/node-exporter           ClusterIP   None             <none>        9100/TCP                              6m38s monitoring    service/prometheus-adapter      ClusterIP   10.243.188.73    <none>        443/TCP                               5m36s monitoring    service/prometheus-k8s          NodePort    10.243.127.141   <none>        9090:35230/TCP,8080:31893/TCP         5m58s monitoring    service/prometheus-operated     ClusterIP   None             <none>        9090/TCP                              4m28s monitoring    service/prometheus-operator     ClusterIP   None             <none>        8443/TCP                              5m20s   NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE kube-system   deployment.apps/coredns                  1/1     1            1           32h kube-system   deployment.apps/local-path-provisioner   1/1     1            1           3h57m kube-system   deployment.apps/metrics-server           1/1     1            1           32h monitoring    deployment.apps/blackbox-exporter        1/1     1            1           7m36s monitoring    deployment.apps/grafana                  1/1     1            1           7m9s monitoring    deployment.apps/kube-state-metrics       1/1     1            1           6m59s monitoring    deployment.apps/prometheus-adapter       2/2     2            2           5m45s monitoring    deployment.apps/prometheus-operator      1/1     1            1           5m28s   NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE kube-system   replicaset.apps/coredns-56f6fc8fd7                  1         1         1       32h kube-system   replicaset.apps/local-path-provisioner-5cf85fd84d   1         1         1       3h57m kube-system   replicaset.apps/metrics-server-5985cbc9d7           1         1         1       32h monitoring    replicaset.apps/blackbox-exporter-d7779b7d4         1         1         1       7m36s monitoring    replicaset.apps/grafana-778555f685                  1         1         1       7m8s monitoring    replicaset.apps/kube-state-metrics-74f55cf6d9       1         1         1       6m59s monitoring    replicaset.apps/prometheus-adapter-784f566c54       2         2         2       5m44s monitoring    replicaset.apps/prometheus-operator-6c55f986bc      1         1         1       5m27s   NAMESPACE    NAME                                 READY   AGE db           statefulset.apps/mysql               1/1     172m monitoring   statefulset.apps/alertmanager-main   3/3     4m30s monitoring   statefulset.apps/prometheus-k8s      2/2     4m29s   NAMESPACE    NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE monitoring   daemonset.apps/node-exporter   5         5         5       5            5           kubernetes.io/os=linux   6m44s   NAMESPACE         NAME                                                             DATA   AGE db                configmap/kube-root-ca.crt                                       1      28h default           configmap/kube-root-ca.crt                                       1      32h kube-node-lease   configmap/kube-root-ca.crt                                       1      32h kube-public       configmap/kube-root-ca.crt                                       1      32h kube-system       configmap/cluster-dns                                            2      32h kube-system       configmap/coredns                                                2      32h kube-system       configmap/extension-apiserver-authentication                     6      32h kube-system       configmap/kube-apiserver-legacy-service-account-token-tracking   1      32h kube-system       configmap/kube-root-ca.crt                                       1      32h kube-system       configmap/local-path-config                                      4      3h57m monitoring        configmap/adapter-config                                         1      5m46s monitoring        configmap/blackbox-exporter-configuration                        1      7m36s monitoring        configmap/grafana-dashboard-alertmanager-overview                1      7m31s monitoring        configmap/grafana-dashboard-apiserver                            1      7m30s monitoring        configmap/grafana-dashboard-cluster-total                        1      7m30s monitoring        configmap/grafana-dashboard-controller-manager                   1      7m29s monitoring        configmap/grafana-dashboard-grafana-overview                     1      7m28s monitoring        configmap/grafana-dashboard-k8s-resources-cluster                1      7m27s monitoring        configmap/grafana-dashboard-k8s-resources-multicluster           1      7m26s monitoring        configmap/grafana-dashboard-k8s-resources-namespace              1      7m26s monitoring        configmap/grafana-dashboard-k8s-resources-node                   1      7m25s monitoring        configmap/grafana-dashboard-k8s-resources-pod                    1      7m25s monitoring        configmap/grafana-dashboard-k8s-resources-workload               1      7m24s monitoring        configmap/grafana-dashboard-k8s-resources-workloads-namespace    1      7m23s monitoring        configmap/grafana-dashboard-kubelet                              1      7m23s monitoring        configmap/grafana-dashboard-namespace-by-pod                     1      7m22s monitoring        configmap/grafana-dashboard-namespace-by-workload                1      7m21s monitoring        configmap/grafana-dashboard-node-cluster-rsrc-use                1      7m21s monitoring        configmap/grafana-dashboard-node-rsrc-use                        1      7m20s monitoring        configmap/grafana-dashboard-nodes                                1      7m17s monitoring        configmap/grafana-dashboard-nodes-aix                            1      7m19s monitoring        configmap/grafana-dashboard-nodes-darwin                         1      7m18s monitoring        configmap/grafana-dashboard-persistentvolumesusage               1      7m16s monitoring        configmap/grafana-dashboard-pod-total                            1      7m15s monitoring        configmap/grafana-dashboard-prometheus                           1      7m14s monitoring        configmap/grafana-dashboard-prometheus-remote-write              1      7m15s monitoring        configmap/grafana-dashboard-proxy                                1      7m12s monitoring        configmap/grafana-dashboard-scheduler                            1      7m11s monitoring        configmap/grafana-dashboard-workload-total                       1      7m11s monitoring        configmap/grafana-dashboards                                     1      7m10s monitoring        configmap/kube-root-ca.crt                                       1      8m13s monitoring        configmap/prometheus-k8s-rulefiles-0                             8      4m35s   NAMESPACE     NAME                                                       TYPE                DATA   AGE db            secret/mysql-secret                                        Opaque              1      26h kube-system   secret/k3s-master01.node-password.k3s                      Opaque              1      32h kube-system   secret/k3s-master02.node-password.k3s                      Opaque              1      32h kube-system   secret/k3s-master03.node-password.k3s                      Opaque              1      32h kube-system   secret/k3s-node01.node-password.k3s                        Opaque              1      32h kube-system   secret/k3s-node02.node-password.k3s                        Opaque              1      32h kube-system   secret/k3s-serving                                         kubernetes.io/tls   2      32h monitoring    secret/alertmanager-main                                   Opaque              1      7m39s monitoring    secret/alertmanager-main-generated                         Opaque              1      4m35s monitoring    secret/alertmanager-main-tls-assets-0                      Opaque              0      4m33s monitoring    secret/alertmanager-main-web-config                        Opaque              1      4m32s monitoring    secret/grafana-config                                      Opaque              1      7m31s monitoring    secret/grafana-datasources                                 Opaque              1      7m31s monitoring    secret/prometheus-k8s                                      Opaque              1      4m33s monitoring    secret/prometheus-k8s-thanos-prometheus-http-client-file   Opaque              1      4m30s monitoring    secret/prometheus-k8s-tls-assets-0                         Opaque              0      4m32s monitoring    secret/prometheus-k8s-web-config                           Opaque              1      4m30s   NAMESPACE    NAME                                                  POD-SELECTOR                                                                                                                                             AGE monitoring   networkpolicy.networking.k8s.io/alertmanager-main     app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus   7m42s monitoring   networkpolicy.networking.k8s.io/blackbox-exporter     app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus                                  7m36s monitoring   networkpolicy.networking.k8s.io/grafana               app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus                                             7m9s monitoring   networkpolicy.networking.k8s.io/kube-state-metrics    app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus                                 6m59s monitoring   networkpolicy.networking.k8s.io/node-exporter         app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus                                      6m44s monitoring   networkpolicy.networking.k8s.io/prometheus-adapter    app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus                          5m43s monitoring   networkpolicy.networking.k8s.io/prometheus-k8s        app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus        6m34s monitoring   networkpolicy.networking.k8s.io/prometheus-operator   app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus                              5m26s   NAMESPACE   NAME                                               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE             storageclass.storage.k8s.io/local-path (default)   rancher.io/local-path   Retain          WaitForFirstConsumer   false                  3h57m   NAMESPACE   NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM         STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE             persistentvolume/pvc-4a232f33-602b-490f-a14d-4fa894ed6bda   1Gi        RWO            Retain           Bound      db/pxc-data   local-path     <unset>                          177m             persistentvolume/pvc-c9c60b30-06f5-4eb8-b0a8-73539a358f7c   1Gi        RWO            Retain           Released   db/pxc-data   local-path     <unset>                          3h22m             persistentvolume/pvc-f522f581-b575-4568-a9ff-1ba85d9c4fa2   1Gi        RWO            Retain           Released   db/pxc-data   local-path     <unset>                          3h43m   NAMESPACE   NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE db          persistentvolumeclaim/pxc-data   Bound    pvc-4a232f33-602b-490f-a14d-4fa894ed6bda   1Gi        RWO            local-path     <unset>                 179m       $ cat /etc/cni/net.d/10-calico.conflist {   "name": "k8s-pod-network",   "cniVersion": "0.3.1",   "plugins": [     {       "type": "calico",       "log_level": "info",       "log_file_path": "/var/log/calico/cni/cni.log",       "datastore_type": "kubernetes",       "nodename": "k3s-master01",       "mtu": 0,       "ipam": {           "type": "calico-ipam",           "container_setings": {"allow_ip_frowarding": true}       },       "policy": {           "type": "k8s"       },       "kubernetes": {           "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"       }     },     {       "type": "portmap",       "snat": true,       "capabilities": {"portMappings": true}     },     {       "type": "bandwidth",       "capabilities": {"bandwidth": true}     }   ] }

九、其他问题

参考文档：
1：https://github.com/percona-lab/percona-docker/
2：https://hub.docker.com/r/percona/percona-xtradb-cluster
3：https://github.com/prometheus-operator/kube-prometheus/
4：https://mp.weixin.qq.com/s/R88DraaaS3bpm3PurzpP9g
5：https://docs.tigera.io/calico/latest/getting-started/kubernetes/k3s/quickstart
6：https://docs.k3s.io/zh/cli/certificate
7：https://docs.rancher.cn/docs/k3s/storage/_index