手把手教你一套完善且高效的k8s离线部署方案-京东云开发者社区

<p># 背景

面对更多项目现场交付，偶而会遇到客户环境不具备公网条件，完全内网部署，这就需要有一套完善且高效的离线部署方案。

# 系统资源

| 编号 | 主机名称 | IP | 资源类型 | CPU | 内存 | 磁盘 |
| --- | --- | --- | --- | --- | --- | --- |
| 01 | k8s-master1 | 10.132.10.91 | CentOS-7 | 4c | 8g | 40g |
| 02 | k8s-master1 | 10.132.10.92 | CentOS-7 | 4c | 8g | 40g |
| 03 | k8s-master1 | 10.132.10.93 | CentOS-7 | 4c | 8g | 40g |
| 04 | k8s-worker1 | 10.132.10.94 | CentOS-7 | 8c | 16g | 200g |
| 05 | k8s-worker2 | 10.132.10.95 | CentOS-7 | 8c | 16g | 200g |
| 06 | k8s-worker3 | 10.132.10.96 | CentOS-7 | 8c | 16g | 200g |
| 07 | k8s-worker4 | 10.132.10.97 | CentOS-7 | 8c | 16g | 200g |
| 08 | k8s-worker5 | 10.132.10.98 | CentOS-7 | 8c | 16g | 200g |
| 09 | k8s-worker6 | 10.132.10.99 | CentOS-7 | 8c | 16g | 200g |
| 10 | k8s-harbor&amp;deploy | 10.132.10.100 | CentOS-7 | 4c | 8g | 500g |
| 11 | k8s-nfs | 10.132.10.101 | CentOS-7 | 2c | 4g | 2000g |
| 12 | k8s-lb | 10.132.10.120 | lb内网 | 2c | 4g | 40g |

# 参数配置

注：在全部节点执行以下操作

## 系统基础设置

工作、日志及数据存储目录设定

```
$ mkdir -p /export/servers
$ mkdir -p /export/logs
$ mkdir -p /export/data
$ mkdir -p /export/upload
```

内核及网络参数优化

```
$ vim /etc/sysctl.conf

# 设置以下内容
fs.file-max = 1048576
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 5
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2 
vm.max_map_count = 262144

# 及时生效
sysctl -w vm.max_map_count=262144
```

### ulimit优化

```
$ vim /etc/security/limits.conf

# 设置以下内容
* soft memlock unlimited
* hard memlock unlimited
* soft nproc 102400
* hard nproc 102400
* soft nofile 1048576
* hard nofile 1048576
```

# 基础环境准备

## ansible安装

### **1.** &nbsp;**环境说明**

| **名称** | **说明** |
| --- | --- |
| 操作系统 | CentOS Linux release 7.8.2003 |
| ansible | 2.9.27 |
| 节点 | deploy |

### **2.** &nbsp;**部署说明**

物联管理平台机器数量繁多，需要ansible进行批量操作机器，节省时间，需要从部署节点至其他节点root免密。

```
注：在不知道root密码情况下，可以手动操作名密，按以下操作步骤执行：
# 需要在部署机器上执行以下命令生成公钥
$ ssh-keygen -t rsa
# 复制~/.ssh/id_rsa.pub内容，并粘贴至其他节点~/.ssh/authorized_keys文件里面
# 如果没有authorized_keys文件，可先执行创建创建在进行粘贴操作
$ touch ~/.ssh/authorized_keys
```

### **3\. 部署步骤**

**1）** &nbsp;**在线安装**

```
$ yum -y install https://releases.ansible.com/ansible/rpm/release/epel-7-x86_64/ansible-2.9.27-1.el7.ans.noarch.rpm
```

**2）** &nbsp;**离线安装**

```
# 提前上传ansible及所有依赖rpm包，并切换至rpm包目录
$ yum -y ./*rpm
```

**3）** &nbsp;**查看版本**

```
$ ansible --version
ansible 2.9.27
 &nbsp;config file = /etc/ansible/ansible.cfg
 &nbsp;configured module search path = [u&#39;/root/.ansible/plugins/modules&#39;, u&#39;/usr/share/ansible/plugins/modules&#39;]
 &nbsp;ansible python module location = /usr/lib/python2.7/site-packages/ansible
 &nbsp;executable location = /usr/bin/ansible
 &nbsp;python version = 2.7.5 (default, Apr &nbsp;2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
```

**4）** &nbsp;**设置管理主机列表**

```
$ vim /etc/ansible/hosts

[master]
10.132.10.91 &nbsp; &nbsp;node_name=k8s-master1
10.132.10.92 &nbsp; &nbsp;node_name=k8s-master2
10.132.10.93 &nbsp; &nbsp;node_name=k8s-master3

[worker]
10.132.10.94 &nbsp; &nbsp;node_name=k8s-worker1
10.132.10.95 &nbsp; &nbsp;node_name=k8s-worker2
10.132.10.96 &nbsp; &nbsp;node_name=k8s-worker3
10.132.10.97 &nbsp; &nbsp;node_name=k8s-worker4
10.132.10.98 &nbsp; &nbsp;node_name=k8s-worker5
10.132.10.99 &nbsp; &nbsp;node_name=k8s-worker6

[etcd]
10.132.10.91 &nbsp; &nbsp;etcd_name=etcd1
10.132.10.92 &nbsp; &nbsp;etcd_name=etcd2
10.132.10.93 &nbsp; &nbsp;etcd_name=etcd3

[k8s:children]
master
worker
```

**5）** &nbsp;**禁用ssh主机检查**

```
$ vi /etc/ansible/ansible.cfg
# 修改以下设置
# uncomment this to disable SSH key host checking
host_key_checking = False
```

**6）** &nbsp;**取消SELINUX设定及放开防火墙**

```
$ ansible k8s -m command -a &quot;setenforce 0&quot;
$ ansible k8s -m command -a &quot;sed --follow-symlinks -i &#39;s/SELINUX=enforcing/SELINUX=disabled/g&#39; /etc/selinux/config&quot;
$ ansible k8s -m command -a &quot;firewall-cmd --set-default-zone=trusted&quot;
$ ansible k8s -m command -a &quot;firewall-cmd --complete-reload&quot;
$ ansible k8s -m command -a &quot;swapoff -a&quot;
```

**7）hosts设置**

```
$ cd /export/upload &amp;&amp; vim hosts_set.sh
#设置以下脚本内容
#!/bin/bashcat &gt; /etc/hosts &lt;&lt; EOF
127.0.0.1 &nbsp; localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 &nbsp; &nbsp; &nbsp; &nbsp; localhost localhost.localdomain localhost6 localhost6.localdomain6
10.132.10.100 deploy harbor
10.132.10.91 master01
10.132.10.92 master02
10.132.10.93 master03
10.132.10.94 worker01
10.132.10.95 worker02
10.132.10.96 worker03
10.132.10.97 worker04
10.132.10.98 worker05
10.132.10.99 worker06
EOF

$ ansible new_worker -m copy -a &#39;src=/export/upload/hosts_set.sh dest=/export/upload&#39;
$ ansible new_worker -m command -a &#39;sh /export/upload/hosts_set.sh&#39;
```

## docker安装

### **1\. 环境说明**

| **名称** | **说明** |
| --- | --- |
| **操作系统** | CentOS Linux release 7.8.2003 |
| **docker** | docker-ce-20.10.17 |
| **节点** | deploy |

### **2\. 部署说明**

### **3\. 部署方法**

**1）** &nbsp;**在线安装**

```
$ yum -y install docker-ce-20.10.17
```

**2）** &nbsp;**离线安装**

```
# 提前上传docker及所有依赖rpm包，并切换至rpm包目录
$ yum -y ./*rpm
```

**3）** &nbsp;**重新加载配置文件，启动并查看状态**

```
$ systemctl start docker
$ systemctl status docker
```

**4）** &nbsp;**设置开机自启**

```
$ systemctl enable docker 
```

**5）** &nbsp;**查看版本**

```
$ docker version
Client: Docker Engine - Community
 Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 20.10.17 
 API version: &nbsp; &nbsp; &nbsp; 1.41 
 Go version: &nbsp; &nbsp; &nbsp; &nbsp;go1.17.11 
 Git commit: &nbsp; &nbsp; &nbsp; &nbsp;100c701 
 Built: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Mon Jun &nbsp;6 23:05:12 2022 
 OS/Arch: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; linux/amd64 
 Context: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; default 
 Experimental: &nbsp; &nbsp; &nbsp;true
 
Server: Docker Engine - Community
 Engine: &nbsp;
 &nbsp;Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;20.10.17 &nbsp;
 &nbsp;API version: &nbsp; &nbsp; &nbsp;1.41 (minimum version 1.12) &nbsp;
 &nbsp;Go version: &nbsp; &nbsp; &nbsp; go1.17.11 &nbsp;
 &nbsp;Git commit: &nbsp; &nbsp; &nbsp; a89b842 &nbsp;
 &nbsp;Built: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Mon Jun &nbsp;6 23:03:33 2022 &nbsp;
 &nbsp;OS/Arch: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;linux/amd64 &nbsp;
 &nbsp;Experimental: &nbsp; &nbsp; false 
 containerd: &nbsp;
 &nbsp;Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1.6.8 &nbsp;
 &nbsp;GitCommit: &nbsp; &nbsp; &nbsp; &nbsp;9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6 
 runc: 
 &nbsp;Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1.1.4 &nbsp;
 &nbsp;GitCommit: &nbsp; &nbsp; &nbsp; &nbsp;v1.1.4-0-g5fd4c4d 
 docker-init: &nbsp;
 Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.19.0 &nbsp;
 GitCommit: &nbsp; &nbsp; &nbsp; &nbsp;de40ad0
```

## docker-compose安装

### **1\. 环境说明**

| **名称** | **说明** |
| --- | --- |
| **操作系统** | CentOS Linux release 7.8.2003 |
| **docker-compose** | docker-compose-linux-x86_64 |
| **节点** | deploy |

### **2\. 部署说明**

harbor私有镜像库依赖。

### **3\. 部署方法**

**1）** &nbsp;**下载docker-compose并上传至服务器**

```
$ curl -L https://github.com/docker/compose/releases/download/v2.9.0/docker-compose-linux-x86_64 -o docker-compose
```

**2）** &nbsp;**修改docker-compose执行权限**

```
$ mv docker-compose /usr/local/bin/
$ chmod +x /usr/local/bin/docker-compose
$ docker-compose version
```

**3）** &nbsp;**查看版本**

```
$ docker-compose version
Docker Compose version v2.9.0
```

## harbor安装

### **1\. 环境说明**

| **名称** | **说明** |
| --- | --- |
| **操作系统** | CentOS Linux release 7.8.2003 |
| **harbor** | harbor-offline-installer-v2.4.3 |
| **节点** | harbor |

### **2\. 部署说明**

私有镜像仓库。

### **3\. 下载harbor离线安装包并上传至服务器**

```
$ wget https://github.com/goharbor/harbor/releases/download/v2.4.3/harbor-offline-installer-v2.4.3.tgz
```

### **4\. 解压安装包**

```
$ tar -xzvf harbor-offline-installer-v2.4.3.tgz -C /export/servers/
$ cd /export/servers/harbor
```

### **5\. 修改配置文件**

```
$ mv harbor.yml.tmpl harbor.yml
$ vim harbor.yml
```

### **6\. 设置以下内容**

```
hostname: 10.132.10.100
http.port: 8090
data_volume: /export/data/harbor
log.location: /export/logs/harbor
```

### **7\. 导入harbor镜像**

```
$ docker load -i harbor.v2.4.3.tar.gz 
# 等待导入harbor依赖镜像文件
$ docker images
REPOSITORY &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TAG &nbsp; &nbsp; &nbsp; IMAGE ID &nbsp; &nbsp; &nbsp; CREATED &nbsp; &nbsp; &nbsp; SIZE
goharbor/harbor-exporter &nbsp; &nbsp; &nbsp; &nbsp;v2.4.3 &nbsp; &nbsp;776ac6ee91f4 &nbsp; 4 weeks ago &nbsp; 81.5MB
goharbor/chartmuseum-photon &nbsp; &nbsp; v2.4.3 &nbsp; &nbsp;f39a9694988d &nbsp; 4 weeks ago &nbsp; 172MB
goharbor/redis-photon &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; v2.4.3 &nbsp; &nbsp;b168e9750dc8 &nbsp; 4 weeks ago &nbsp; 154MB
goharbor/trivy-adapter-photon &nbsp; v2.4.3 &nbsp; &nbsp;a406a715461c &nbsp; 4 weeks ago &nbsp; 251MB
goharbor/notary-server-photon &nbsp; v2.4.3 &nbsp; &nbsp;da89404c7cf9 &nbsp; 4 weeks ago &nbsp; 109MB
goharbor/notary-signer-photon &nbsp; v2.4.3 &nbsp; &nbsp;38468ac13836 &nbsp; 4 weeks ago &nbsp; 107MB
goharbor/harbor-registryctl &nbsp; &nbsp; v2.4.3 &nbsp; &nbsp;61243a84642b &nbsp; 4 weeks ago &nbsp; 135MB
goharbor/registry-photon &nbsp; &nbsp; &nbsp; &nbsp;v2.4.3 &nbsp; &nbsp;9855479dd6fa &nbsp; 4 weeks ago &nbsp; 77.9MB
goharbor/nginx-photon &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; v2.4.3 &nbsp; &nbsp;0165c71ef734 &nbsp; 4 weeks ago &nbsp; 44.4MB
goharbor/harbor-log &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; v2.4.3 &nbsp; &nbsp;57ceb170dac4 &nbsp; 4 weeks ago &nbsp; 161MB
goharbor/harbor-jobservice &nbsp; &nbsp; &nbsp;v2.4.3 &nbsp; &nbsp;7fea87c4b884 &nbsp; 4 weeks ago &nbsp; 219MB
goharbor/harbor-core &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;v2.4.3 &nbsp; &nbsp;d864774a3b8f &nbsp; 4 weeks ago &nbsp; 197MB
goharbor/harbor-portal &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;v2.4.3 &nbsp; &nbsp;85f00db66862 &nbsp; 4 weeks ago &nbsp; 53.4MB
goharbor/harbor-db &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;v2.4.3 &nbsp; &nbsp;7693d44a2ad6 &nbsp; 4 weeks ago &nbsp; 225MB
goharbor/prepare &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;v2.4.3 &nbsp; &nbsp;c882d74725ee &nbsp; 4 weeks ago &nbsp; 268MB
```

### **8\. 启动harbor**

```
./prepare &nbsp;# 如果有二次修改harbor.yml文件，请执行使配置文件生效
./install.sh --help # 查看启动参数
./install.sh --with-chartmuseum
```

# 运行环境搭建

## docker安装

### **1\. 环境说明**

| **名称** | **说明** |
| --- | --- |
| **操作系统** | CentOS Linux release 7.8.2003 |
| **docker** | docker-ce-20.10.17 |
| **节点** | k8s集群全部节点 |

### **2\. 部署说明**

k8s容器运行环境docker部署

### **3\. 部署方法**

**1）** &nbsp;**上传docker及依赖rpm包**

```
$ ls /export/upload/docker-rpm.tgz 
```

**2）** &nbsp;**分发安装包**

```
$ ansible k8s -m copy -a &quot;src=/export/upload/docker-rpm.tgz dest=/export/upload/&quot;
# 全部节点返回以下信息
CHANGED =&gt; {
 &nbsp; &nbsp;&quot;ansible_facts&quot;: {
 &nbsp; &nbsp; &nbsp; &nbsp;&quot;discovered_interpreter_python&quot;: &quot;/usr/bin/python&quot;
 &nbsp; &nbsp;}, 
 &nbsp; &nbsp;&quot;changed&quot;: true, 
 &nbsp; &nbsp;&quot;checksum&quot;: &quot;acd3897edb624cd18a197bcd026e6769797f4f05&quot;, 
 &nbsp; &nbsp;&quot;dest&quot;: &quot;/export/upload/docker-rpm.tgz&quot;, 
 &nbsp; &nbsp;&quot;gid&quot;: 0, 
 &nbsp; &nbsp;&quot;group&quot;: &quot;root&quot;, 
 &nbsp; &nbsp;&quot;md5sum&quot;: &quot;3ba6d9fe6b2ac70860b6638b88d3c89d&quot;, 
 &nbsp; &nbsp;&quot;mode&quot;: &quot;0644&quot;, 
 &nbsp; &nbsp;&quot;owner&quot;: &quot;root&quot;, 
 &nbsp; &nbsp;&quot;secontext&quot;: &quot;system_u:object_r:usr_t:s0&quot;, 
 &nbsp; &nbsp;&quot;size&quot;: 103234394, 
 &nbsp; &nbsp;&quot;src&quot;: &quot;/root/.ansible/tmp/ansible-tmp-1661836788.82-13591-17885284311930/source&quot;, 
 &nbsp; &nbsp;&quot;state&quot;: &quot;file&quot;, 
 &nbsp; &nbsp;&quot;uid&quot;: 0
}
```

**3）** &nbsp;**执行解压并安装**

```
$ ansible k8s -m shell -a &quot;tar xzvf /export/upload/docker-rpm.tgz -C /export/upload/ &amp;&amp; yum -y install /export/upload/docker-rpm/*&quot;
```

**4）** &nbsp;**设置开机自启并启动**

```
$ ansible k8s -m shell -a &quot;systemctl enable docker &amp;&amp; systemctl start docker&quot;
```

**5）** &nbsp;**查看版本**

```
$ ansible k8s -m shell -a &quot;docker version&quot;
# 全部节点返回以下信息
CHANGED | rc=0 &gt;&gt;
Client: Docker Engine - Community
 Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 20.10.17
 API version: &nbsp; &nbsp; &nbsp; 1.41
 Go version: &nbsp; &nbsp; &nbsp; &nbsp;go1.17.11
 Git commit: &nbsp; &nbsp; &nbsp; &nbsp;100c701
 Built: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Mon Jun &nbsp;6 23:05:12 2022
 OS/Arch: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; linux/amd64
 Context: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; default
 Experimental: &nbsp; &nbsp; &nbsp;true

Server: Docker Engine - Community
 Engine:
 &nbsp;Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;20.10.17
 &nbsp;API version: &nbsp; &nbsp; &nbsp;1.41 (minimum version 1.12)
 &nbsp;Go version: &nbsp; &nbsp; &nbsp; go1.17.11
 &nbsp;Git commit: &nbsp; &nbsp; &nbsp; a89b842
 &nbsp;Built: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Mon Jun &nbsp;6 23:03:33 2022
 &nbsp;OS/Arch: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;linux/amd64
 &nbsp;Experimental: &nbsp; &nbsp; false
 containerd:
 &nbsp;Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1.6.8
 &nbsp;GitCommit: &nbsp; &nbsp; &nbsp; &nbsp;9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
 &nbsp;Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1.1.4
 &nbsp;GitCommit: &nbsp; &nbsp; &nbsp; &nbsp;v1.1.4-0-g5fd4c4d
 docker-init:
 &nbsp;Version: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.19.0
 &nbsp;GitCommit: &nbsp; &nbsp; &nbsp; &nbsp;de40ad0
```

## kubernetes安装

**有网环境安装**

```
# 添加阿里云YUM的软件源：
cat &gt; /etc/yum.repos.d/kubernetes.repo &lt;&lt; EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
```

**下载离线安装包**

```
# 创建rpm软件存储目录：
mkdir -p /export/download/kubeadm-rpm

# 执行命令：
yum install -y kubelet-1.22.4 kubeadm-1.22.4 kubectl-1.22.4 --downloadonly --downloaddir /export/download/kubeadm-rpm
```

**无网环境安装**

**1）** &nbsp;**上传kubeadm及依赖rpm包**

```
$ ls /export/upload/
kubeadm-rpm.tgz 
```

**2）** &nbsp;**分发安装包**

```
$ ansible k8s -m copy -a &quot;src=/export/upload/kubeadm-rpm.tgz dest=/export/upload/&quot;
# 全部节点返回以下信息
CHANGED =&gt; {
 &nbsp; &nbsp;&quot;ansible_facts&quot;: {
 &nbsp; &nbsp; &nbsp; &nbsp;&quot;discovered_interpreter_python&quot;: &quot;/usr/bin/python&quot;
 &nbsp; &nbsp;}, 
 &nbsp; &nbsp;&quot;changed&quot;: true, 
 &nbsp; &nbsp;&quot;checksum&quot;: &quot;3fe96fe1aa7f4a09d86722f79f36fb8fde69facb&quot;, 
 &nbsp; &nbsp;&quot;dest&quot;: &quot;/export/upload/kubeadm-rpm.tgz&quot;, 
 &nbsp; &nbsp;&quot;gid&quot;: 0, 
 &nbsp; &nbsp;&quot;group&quot;: &quot;root&quot;, 
 &nbsp; &nbsp;&quot;md5sum&quot;: &quot;80d5bda420db6ea23ad75dcf0f76e858&quot;, 
 &nbsp; &nbsp;&quot;mode&quot;: &quot;0644&quot;, 
 &nbsp; &nbsp;&quot;owner&quot;: &quot;root&quot;, 
 &nbsp; &nbsp;&quot;secontext&quot;: &quot;system_u:object_r:usr_t:s0&quot;, 
 &nbsp; &nbsp;&quot;size&quot;: 67423355, 
 &nbsp; &nbsp;&quot;src&quot;: &quot;/root/.ansible/tmp/ansible-tmp-1661840257.4-33361-139823848282879/source&quot;, 
 &nbsp; &nbsp;&quot;state&quot;: &quot;file&quot;, 
 &nbsp; &nbsp;&quot;uid&quot;: 0
}
```

**3）** &nbsp;**执行解压并安装**

```
$ ansible k8s -m shell -a &quot;tar xzvf /export/upload/kubeadm-rpm.tgz -C /export/upload/ &amp;&amp; yum -y install /export/upload/kubeadm-rpm/*&quot;
```

**4）** &nbsp;**设置开机自启并启动**

```
$ ansible k8s -m shell -a &quot;systemctl enable kubelet &amp;&amp; systemctl start kubelet&quot;
注：此时kubelet启动失败，会进入不断重启，这个是正常现象，执行init或join后问题会自动解决，对此官网有如下描述，也就是此时不用理会kubelet.service，可执行发下命令查看kubelet状态。
$ journalctl -xefu kubelet
```

**4）** &nbsp;**分发依赖镜像至集群节点**

```
# 可以在有公网环境提前下载镜像
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.4
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.4
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.4
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.4
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5
$ docker pull rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
$ docker pull rancher/mirrored-flannelcni-flannel:v0.19.1
# 导出镜像文件，上传部署节点并导入镜像库
$ ls /export/upload

$ docker load -i google_containers-coredns-v1.8.4.tar
$ docker load -i google_containers-etcd:3.5.0-0.tar
$ docker load -i google_containers-kube-apiserver:v1.22.4.tar
$ docker load -i google_containers-kube-controller-manager-v1.22.4.tar
$ docker load -i google_containers-kube-proxy-v1.22.4.tar
$ docker load -i google_containers-kube-scheduler-v1.22.4.tar
$ docker load -i google_containers-pause-3.5.tar
$ docker load -i rancher-mirrored-flannelcni-flannel-cni-plugin-v1.1.0.tar
$ docker load -i rancher-mirrored-flannelcni-flannel-v0.19.1.tar

# 镜像打harbor镜像库tag
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4 10.132.10.100:8090/community/coredns:v1.8.4
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0 10.132.10.100:8090/community/etcd:3.5.0-0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.4 10.132.10.100:8090/community/kube-apiserver:v1.22.4
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.4 10.132.10.100:8090/community/kube-controller-manager:v1.22.4
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.4 10.132.10.100:8090/community/kube-proxy:v1.22.4
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.4 10.132.10.100:8090/community/kube-scheduler:v1.22.4
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 10.132.10.100:8090/community/pause:3.5
$ docker tag rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 10.132.10.100:8090/community/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
$ docker tag rancher/mirrored-flannelcni-flannel:v0.19.1 10.132.10.100:8090/community/mirrored-flannelcni-flannel:v0.19.1

# 推送至harbor镜像库
$ docker push 192.168.186.120:8090/community/coredns:v1.8.4
$ docker push 192.168.186.120:8090/community/etcd:3.5.0-0
$ docker push 192.168.186.120:8090/community/kube-apiserver:v1.22.4
$ docker push 192.168.186.120:8090/community/kube-controller-manager:v1.22.4
$ docker push 192.168.186.120:8090/community/kube-proxy:v1.22.4
$ docker push 192.168.186.120:8090/community/kube-scheduler:v1.22.4
$ docker push 192.168.186.120:8090/community/pause:3.5
$ docker push 192.168.186.120:8090/community/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
$ docker push 192.168.186.120:8090/community/mirrored-flannelcni-flannel:v0.19.1
```

**5）部署首个master**

```
$ kubeadm init \
--control-plane-endpoint &quot;10.132.10.91:6443&quot; \
--image-repository 10.132.10.100/community \
--kubernetes-version v1.22.4 \
--service-cidr=172.16.0.0/16 \
--pod-network-cidr=10.244.0.0/16 \
--token &quot;abcdef.0123456789abcdef&quot; \
--token-ttl &quot;0&quot; \
--upload-certs

# 显示以下信息
[init] Using Kubernetes version: v1.22.4
[preflight] Running pre-flight checks
	[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using &#39;kubeadm config images pull&#39;
[certs] Using certificateDir folder &quot;/etc/kubernetes/pki&quot;
[certs] Generating &quot;ca&quot; certificate and key
[certs] Generating &quot;apiserver&quot; certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master01] and IPs [172.16.0.1 10.132.10.91]
[certs] Generating &quot;apiserver-kubelet-client&quot; certificate and key
[certs] Generating &quot;front-proxy-ca&quot; certificate and key
[certs] Generating &quot;front-proxy-client&quot; certificate and key
[certs] Generating &quot;etcd/ca&quot; certificate and key
[certs] Generating &quot;etcd/server&quot; certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master01] and IPs [10.132.10.91 127.0.0.1 ::1]
[certs] Generating &quot;etcd/peer&quot; certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master01] and IPs [10.132.10.91 127.0.0.1 ::1]
[certs] Generating &quot;etcd/healthcheck-client&quot; certificate and key
[certs] Generating &quot;apiserver-etcd-client&quot; certificate and key
[certs] Generating &quot;sa&quot; key and public key
[kubeconfig] Using kubeconfig folder &quot;/etc/kubernetes&quot;
[kubeconfig] Writing &quot;admin.conf&quot; kubeconfig file
[kubeconfig] Writing &quot;kubelet.conf&quot; kubeconfig file
[kubeconfig] Writing &quot;controller-manager.conf&quot; kubeconfig file
[kubeconfig] Writing &quot;scheduler.conf&quot; kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file &quot;/var/lib/kubelet/kubeadm-flags.env&quot;
[kubelet-start] Writing kubelet configuration to file &quot;/var/lib/kubelet/config.yaml&quot;
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder &quot;/etc/kubernetes/manifests&quot;
[control-plane] Creating static Pod manifest for &quot;kube-apiserver&quot;
[control-plane] Creating static Pod manifest for &quot;kube-controller-manager&quot;
[control-plane] Creating static Pod manifest for &quot;kube-scheduler&quot;
[etcd] Creating static Pod manifest for local etcd in &quot;/etc/kubernetes/manifests&quot;
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory &quot;/etc/kubernetes/manifests&quot;. This can take up to 4m0s
[apiclient] All control plane components are healthy after 11.008638 seconds
[upload-config] Storing the configuration used in ConfigMap &quot;kubeadm-config&quot; in the &quot;kube-system&quot; Namespace
[kubelet] Creating a ConfigMap &quot;kubelet-config-1.22&quot; in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret &quot;kubeadm-certs&quot; in the &quot;kube-system&quot; Namespace
[upload-certs] Using certificate key:
9151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13
[mark-control-plane] Marking the node master01 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the &quot;cluster-info&quot; ConfigMap in the &quot;kube-public&quot; namespace
[kubelet-finalize] Updating &quot;/etc/kubernetes/kubelet.conf&quot; to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

&nbsp;mkdir -p $HOME/.kube
 &nbsp;sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 &nbsp;sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

&nbsp;export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run &quot;kubectl apply -f [podnetwork].yaml&quot; with one of the options listed at:
 &nbsp;https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

&nbsp;kubeadm join 10.132.10.91:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2 \
	--control-plane --certificate-key 9151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
&quot;kubeadm init phase upload-certs --upload-certs&quot; to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.132.10.91:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2
```

**6）生成kubelet环境配置文件**

```
# 执行命令
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

**7）配置网络插件flannel**

```
# 创建flannel.yml文件
$ touch /export/servers/kubernetes/flannel.yml
$ vim /export/servers/kubernetes/flannel.yml
# 设置以下内容，需要关注有网无网时对应的地址切换
---
kind: Namespace
apiVersion: v1
metadata:
 &nbsp;name: kube-flannel
 &nbsp;labels:
 &nbsp; &nbsp;pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 &nbsp;name: flannel
rules:
- apiGroups:
 &nbsp;- &quot;&quot;
 &nbsp;resources:
 &nbsp;- pods
 &nbsp;verbs:
 &nbsp;- get
- apiGroups:
 &nbsp;- &quot;&quot;
 &nbsp;resources:
 &nbsp;- nodes
 &nbsp;verbs:
 &nbsp;- list
 &nbsp;- watch
- apiGroups:
 &nbsp;- &quot;&quot;
 &nbsp;resources:
 &nbsp;- nodes/status
 &nbsp;verbs:
 &nbsp;- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 &nbsp;name: flannel
roleRef:
 &nbsp;apiGroup: rbac.authorization.k8s.io
 &nbsp;kind: ClusterRole
 &nbsp;name: flannel
subjects:
- kind: ServiceAccount
 &nbsp;name: flannel
 &nbsp;namespace: kube-flannel
---
apiVersion: v1
kind: ServiceAccount
metadata:
 &nbsp;name: flannel
 &nbsp;namespace: kube-flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
 &nbsp;name: kube-flannel-cfg
 &nbsp;namespace: kube-flannel
 &nbsp;labels:
 &nbsp; &nbsp;tier: node
 &nbsp; &nbsp;app: flannel
data:
 &nbsp;cni-conf.json: |
 &nbsp; &nbsp;{
 &nbsp; &nbsp; &nbsp;&quot;name&quot;: &quot;cbr0&quot;,
 &nbsp; &nbsp; &nbsp;&quot;cniVersion&quot;: &quot;0.3.1&quot;,
 &nbsp; &nbsp; &nbsp;&quot;plugins&quot;: [
 &nbsp; &nbsp; &nbsp; &nbsp;{
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;type&quot;: &quot;flannel&quot;,
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;delegate&quot;: {
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;hairpinMode&quot;: true,
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;isDefaultGateway&quot;: true
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;}
 &nbsp; &nbsp; &nbsp; &nbsp;},
 &nbsp; &nbsp; &nbsp; &nbsp;{
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;type&quot;: &quot;portmap&quot;,
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;capabilities&quot;: {
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;portMappings&quot;: true
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;}
 &nbsp; &nbsp; &nbsp; &nbsp;}
 &nbsp; &nbsp; &nbsp;]
 &nbsp; &nbsp;}
 &nbsp;net-conf.json: |
 &nbsp; &nbsp;{
 &nbsp; &nbsp; &nbsp;&quot;Network&quot;: &quot;10.244.0.0/16&quot;,
 &nbsp; &nbsp; &nbsp;&quot;Backend&quot;: {
 &nbsp; &nbsp; &nbsp; &nbsp;&quot;Type&quot;: &quot;vxlan&quot;
 &nbsp; &nbsp; &nbsp;}
 &nbsp; &nbsp;}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
 &nbsp;name: kube-flannel-ds
 &nbsp;namespace: kube-flannel
 &nbsp;labels:
 &nbsp; &nbsp;tier: node
 &nbsp; &nbsp;app: flannel
spec:
 &nbsp;selector:
 &nbsp; &nbsp;matchLabels:
 &nbsp; &nbsp; &nbsp;app: flannel
 &nbsp;template:
 &nbsp; &nbsp;metadata:
 &nbsp; &nbsp; &nbsp;labels:
 &nbsp; &nbsp; &nbsp; &nbsp;tier: node
 &nbsp; &nbsp; &nbsp; &nbsp;app: flannel
 &nbsp; &nbsp;spec:
 &nbsp; &nbsp; &nbsp;affinity:
 &nbsp; &nbsp; &nbsp; &nbsp;nodeAffinity:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;requiredDuringSchedulingIgnoredDuringExecution:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;nodeSelectorTerms:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- matchExpressions:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- key: kubernetes.io/os
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;operator: In
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;values:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- linux
 &nbsp; &nbsp; &nbsp;hostNetwork: true
 &nbsp; &nbsp; &nbsp;priorityClassName: system-node-critical
 &nbsp; &nbsp; &nbsp;tolerations:
 &nbsp; &nbsp; &nbsp;- operator: Exists
 &nbsp; &nbsp; &nbsp; &nbsp;effect: NoSchedule
 &nbsp; &nbsp; &nbsp;serviceAccountName: flannel
 &nbsp; &nbsp; &nbsp;initContainers:
 &nbsp; &nbsp; &nbsp;- name: install-cni-plugin
 &nbsp; &nbsp; &nbsp; &nbsp;# 在有网环境下可以切换下面地址
 &nbsp; &nbsp; &nbsp; &nbsp;# image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
 &nbsp; &nbsp; &nbsp; &nbsp;# 在无网环境下需要使用私有harbor地址
 &nbsp; &nbsp; &nbsp; &nbsp;image: 10.132.10.100:8090/community/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
 &nbsp; &nbsp; &nbsp; &nbsp;command:
 &nbsp; &nbsp; &nbsp; &nbsp;- cp
 &nbsp; &nbsp; &nbsp; &nbsp;args:
 &nbsp; &nbsp; &nbsp; &nbsp;- -f
 &nbsp; &nbsp; &nbsp; &nbsp;- /flannel
 &nbsp; &nbsp; &nbsp; &nbsp;- /opt/cni/bin/flannel
 &nbsp; &nbsp; &nbsp; &nbsp;volumeMounts:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: cni-plugin
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /opt/cni/bin
 &nbsp; &nbsp; &nbsp;- name: install-cni
 &nbsp; &nbsp; &nbsp; &nbsp;# 在有网环境下可以切换下面地址
 &nbsp; &nbsp; &nbsp; &nbsp;# image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.1
 &nbsp; &nbsp; &nbsp; &nbsp;# 在无网环境下需要使用私有harbor地址
 &nbsp; &nbsp; &nbsp; &nbsp;image: 10.132.10.100:8090/community/mirrored-flannelcni-flannel:v0.19.1
 &nbsp; &nbsp; &nbsp; &nbsp;command:
 &nbsp; &nbsp; &nbsp; &nbsp;- cp
 &nbsp; &nbsp; &nbsp; &nbsp;args:
 &nbsp; &nbsp; &nbsp; &nbsp;- -f
 &nbsp; &nbsp; &nbsp; &nbsp;- /etc/kube-flannel/cni-conf.json
 &nbsp; &nbsp; &nbsp; &nbsp;- /etc/cni/net.d/10-flannel.conflist
 &nbsp; &nbsp; &nbsp; &nbsp;volumeMounts:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: cni
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /etc/cni/net.d
 &nbsp; &nbsp; &nbsp; &nbsp;- name: flannel-cfg
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /etc/kube-flannel/
 &nbsp; &nbsp; &nbsp;containers:
 &nbsp; &nbsp; &nbsp;- name: kube-flannel
 &nbsp; &nbsp; &nbsp; &nbsp;# 在有网环境下可以切换下面地址
 &nbsp; &nbsp; &nbsp; &nbsp;# image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.1
 &nbsp; &nbsp; &nbsp; &nbsp;# 在无网环境下需要使用私有harbor地址
 &nbsp; &nbsp; &nbsp; &nbsp;image: 10.132.10.100:8090/community/mirrored-flannelcni-flannel:v0.19.1
 &nbsp; &nbsp; &nbsp; &nbsp;command:
 &nbsp; &nbsp; &nbsp; &nbsp;- /opt/bin/flanneld
 &nbsp; &nbsp; &nbsp; &nbsp;args:
 &nbsp; &nbsp; &nbsp; &nbsp;- --ip-masq
 &nbsp; &nbsp; &nbsp; &nbsp;- --kube-subnet-mgr
 &nbsp; &nbsp; &nbsp; &nbsp;resources:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;requests:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;cpu: &quot;100m&quot;
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;memory: &quot;50Mi&quot;
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;limits:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;cpu: &quot;100m&quot;
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;memory: &quot;50Mi&quot;
 &nbsp; &nbsp; &nbsp; &nbsp;securityContext:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;privileged: false
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;capabilities:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;add: [&quot;NET_ADMIN&quot;, &quot;NET_RAW&quot;]
 &nbsp; &nbsp; &nbsp; &nbsp;env:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: POD_NAME
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;valueFrom:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;fieldRef:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;fieldPath: metadata.name
 &nbsp; &nbsp; &nbsp; &nbsp;- name: POD_NAMESPACE
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;valueFrom:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;fieldRef:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;fieldPath: metadata.namespace
 &nbsp; &nbsp; &nbsp; &nbsp;- name: EVENT_QUEUE_DEPTH
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;value: &quot;5000&quot;
 &nbsp; &nbsp; &nbsp; &nbsp;volumeMounts:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: run
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /run/flannel
 &nbsp; &nbsp; &nbsp; &nbsp;- name: flannel-cfg
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /etc/kube-flannel/
 &nbsp; &nbsp; &nbsp; &nbsp;- name: xtables-lock
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /run/xtables.lock
 &nbsp; &nbsp; &nbsp;volumes:
 &nbsp; &nbsp; &nbsp;- name: run
 &nbsp; &nbsp; &nbsp; &nbsp;hostPath:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;path: /run/flannel
 &nbsp; &nbsp; &nbsp;- name: cni-plugin
 &nbsp; &nbsp; &nbsp; &nbsp;hostPath:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;path: /opt/cni/bin
 &nbsp; &nbsp; &nbsp;- name: cni
 &nbsp; &nbsp; &nbsp; &nbsp;hostPath:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;path: /etc/cni/net.d
 &nbsp; &nbsp; &nbsp;- name: flannel-cfg
 &nbsp; &nbsp; &nbsp; &nbsp;configMap:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;name: kube-flannel-cfg
 &nbsp; &nbsp; &nbsp;- name: xtables-lock
 &nbsp; &nbsp; &nbsp; &nbsp;hostPath:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;path: /run/xtables.lock
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;type: FileOrCreate
```

**8）安装网络插件flannel**

```
# 生效yml配置文件
$ kubectl apply -f kube-flannel.yml

# 查看pods状态
$ kubectl get pods -A
NAMESPACE &nbsp; &nbsp; &nbsp;NAME &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; READY &nbsp; STATUS &nbsp; &nbsp;RESTARTS &nbsp; AGE
kube-flannel &nbsp; kube-flannel-ds-kjmt4 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;148m
kube-system &nbsp; &nbsp;coredns-7f84d7b4b5-7qr8g &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4h18m
kube-system &nbsp; &nbsp;coredns-7f84d7b4b5-fljws &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4h18m
kube-system &nbsp; &nbsp;etcd-master01 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4h19m
kube-system &nbsp; &nbsp;kube-apiserver-master01 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4h19m
kube-system &nbsp; &nbsp;kube-controller-manager-master01 &nbsp; 1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4h19m
kube-system &nbsp; &nbsp;kube-proxy-wzq2t &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4h18m
kube-system &nbsp; &nbsp;kube-scheduler-master01 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4h19m
```

**9）加入其他master节点**

```
# 在master01执行如下操作
# 查看token列表
$ kubeadm token list

# master01执行init操作后生成加入命令如下
$ kubeadm join 10.132.10.91:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2 \
--control-plane --certificate-key 9151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13

# 在其他master节点执行如下操作
# 分别执行上一步的加入命令，加入master节点至集群
$ kubeadm join 10.132.10.91:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2 \
--control-plane --certificate-key 9151ea7bb260a42297f2edc486d5792f67d9868169310b82ef1eb18f6e4c0f13

# 此处如果报错，一般是certificate-key过期，可以在master01执行如下命令更新
$ kubeadm init phase upload-certs --upload-certs
3b647155b06311d39faf70cb094d9a5e102afd1398323e820cfb3cfd868ae58f

# 将上面生成的值替换certificate-key值再次在其他master节点执行如下命令
$ kubeadm join 10.132.10.91:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2 &nbsp; &nbsp; 
--control-plane 
--certificate-key 3b647155b06311d39faf70cb094d9a5e102afd1398323e820cfb3cfd868ae58f

# 生成kubelet环境配置文件
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 在任意master节点执行查看节点状态命令
$ kubectl get nodes
NAME &nbsp; &nbsp; &nbsp; STATUS &nbsp; ROLES &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;AGE &nbsp; &nbsp; VERSION
master01 &nbsp; Ready &nbsp; &nbsp;control-plane,master &nbsp; 5h58m &nbsp; v1.22.4
master02 &nbsp; Ready &nbsp; &nbsp;control-plane,master &nbsp; 45m &nbsp; &nbsp; v1.22.4
master03 &nbsp; Ready &nbsp; &nbsp;control-plane,master &nbsp; 44m &nbsp; &nbsp; v1.22.4
```

**9）加入worker节点**

```
# 在其他worker节点执行master01执行init操作后生成的加入命令如下
# 分别执行上一步的加入命令，加入master节点至集群
$ kubeadm join 10.132.10.91:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:4884a98b0773bc89c36dc5fa51569293103ff093e9124431c4c8c2d5801a96a2

# 此处如果报错，一般是token过期，可以在master01执行如下命令重新生成加入命令
$ kubeadm token create --print-join-command
kubeadm join 10.132.10.91:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:cf30ddd3df1c6215b886df1ea378a68ad5a9faad7933d53ca9891ebbdf9a1c3f

# 将上面生成的加入命令再次在其他worker节点执行
# 查看集成状态
$ kubectl get nodes
NAME &nbsp; &nbsp; &nbsp; STATUS &nbsp; ROLES &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;AGE &nbsp; &nbsp; VERSION
master01 &nbsp; Ready &nbsp; &nbsp;control-plane,master &nbsp; 6h12m &nbsp; v1.22.4
master02 &nbsp; Ready &nbsp; &nbsp;control-plane,master &nbsp; 58m &nbsp; &nbsp; v1.22.4
master03 &nbsp; Ready &nbsp; &nbsp;control-plane,master &nbsp; 57m &nbsp; &nbsp; v1.22.4
worker01 &nbsp; Ready</p><none>5m12s &nbsp; v1.22.4
worker02 &nbsp; Ready<none>4m10s &nbsp; v1.22.4
worker03 &nbsp; Ready<none>3m42s &nbsp; v1.22.4
```

**10）配置kubernetes dashboard**

```
apiVersion: v1
kind: Namespace
metadata:
 &nbsp;name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard
 &nbsp;namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard
 &nbsp;namespace: kubernetes-dashboard
spec:
 &nbsp;type: NodePort
 &nbsp;ports:
 &nbsp; &nbsp;- port: 443
 &nbsp; &nbsp; &nbsp;targetPort: 8443
 &nbsp; &nbsp; &nbsp;nodePort: 31001
 &nbsp;selector:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard-certs
 &nbsp;namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard-key-holder
 &nbsp;namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard-settings
 &nbsp;namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard
 &nbsp;namespace: kubernetes-dashboard
rules:
 &nbsp;# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
 &nbsp;- apiGroups: [&quot;&quot;]
 &nbsp; &nbsp;resources: [&quot;secrets&quot;]
 &nbsp; &nbsp;resourceNames: [&quot;kubernetes-dashboard-key-holder&quot;, &quot;kubernetes-dashboard-certs&quot;, &quot;kubernetes-dashboard-csrf&quot;]
 &nbsp; &nbsp;verbs: [&quot;get&quot;, &quot;update&quot;, &quot;delete&quot;]
 &nbsp; &nbsp;# Allow Dashboard to get and update &#39;kubernetes-dashboard-settings&#39; config map.
 &nbsp;- apiGroups: [&quot;&quot;]
 &nbsp; &nbsp;resources: [&quot;configmaps&quot;]
 &nbsp; &nbsp;resourceNames: [&quot;kubernetes-dashboard-settings&quot;]
 &nbsp; &nbsp;verbs: [&quot;get&quot;, &quot;update&quot;]
 &nbsp; &nbsp;# Allow Dashboard to get metrics.
 &nbsp;- apiGroups: [&quot;&quot;]
 &nbsp; &nbsp;resources: [&quot;services&quot;]
 &nbsp; &nbsp;resourceNames: [&quot;heapster&quot;, &quot;dashboard-metrics-scraper&quot;]
 &nbsp; &nbsp;verbs: [&quot;proxy&quot;]
 &nbsp;- apiGroups: [&quot;&quot;]
 &nbsp; &nbsp;resources: [&quot;services/proxy&quot;]
 &nbsp; &nbsp;resourceNames: [&quot;heapster&quot;, &quot;http:heapster:&quot;, &quot;https:heapster:&quot;, &quot;dashboard-metrics-scraper&quot;, &quot;http:dashboard-metrics-scraper&quot;]
 &nbsp; &nbsp;verbs: [&quot;get&quot;]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard
rules:
 &nbsp;# Allow Metrics Scraper to get metrics from the Metrics server
 &nbsp;- apiGroups: [&quot;metrics.k8s.io&quot;]
 &nbsp; &nbsp;resources: [&quot;pods&quot;, &quot;nodes&quot;]
 &nbsp; &nbsp;verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard
 &nbsp;namespace: kubernetes-dashboard
roleRef:
 &nbsp;apiGroup: rbac.authorization.k8s.io
 &nbsp;kind: Role
 &nbsp;name: kubernetes-dashboard
subjects:
 &nbsp;- kind: ServiceAccount
 &nbsp; &nbsp;name: kubernetes-dashboard
 &nbsp; &nbsp;namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 &nbsp;name: kubernetes-dashboard
roleRef:
 &nbsp;apiGroup: rbac.authorization.k8s.io
 &nbsp;kind: ClusterRole
 &nbsp;name: kubernetes-dashboard
subjects:
 &nbsp;- kind: ServiceAccount
 &nbsp; &nbsp;name: kubernetes-dashboard
 &nbsp; &nbsp;namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;name: kubernetes-dashboard
 &nbsp;namespace: kubernetes-dashboard
spec:
 &nbsp;replicas: 1
 &nbsp;revisionHistoryLimit: 10
 &nbsp;selector:
 &nbsp; &nbsp;matchLabels:
 &nbsp; &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp;template:
 &nbsp; &nbsp;metadata:
 &nbsp; &nbsp; &nbsp;labels:
 &nbsp; &nbsp; &nbsp; &nbsp;k8s-app: kubernetes-dashboard
 &nbsp; &nbsp;spec:
 &nbsp; &nbsp; &nbsp;securityContext:
 &nbsp; &nbsp; &nbsp; &nbsp;seccompProfile:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;type: RuntimeDefault
 &nbsp; &nbsp; &nbsp;containers:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: kubernetes-dashboard
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;image: kubernetesui/dashboard:v2.5.0
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;imagePullPolicy: Always
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ports:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- containerPort: 8443
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;protocol: TCP
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;args:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- --auto-generate-certificates
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- --namespace=kubernetes-dashboard
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Uncomment the following line to manually specify Kubernetes API server Host
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# If not specified, Dashboard will attempt to auto discover the API server and connect
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# to it. Uncomment only if the default does not work.
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# - --apiserver-host=http://my-address:port
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;volumeMounts:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- name: kubernetes-dashboard-certs
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /certs
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Create on-disk volume to store exec logs
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- mountPath: /tmp
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;name: tmp-volume
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;livenessProbe:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;httpGet:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;scheme: HTTPS
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;path: /
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;port: 8443
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;initialDelaySeconds: 30
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;timeoutSeconds: 30
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;securityContext:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;allowPrivilegeEscalation: false
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;readOnlyRootFilesystem: true
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;runAsUser: 1001
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;runAsGroup: 2001
 &nbsp; &nbsp; &nbsp;volumes:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: kubernetes-dashboard-certs
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;secret:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;secretName: kubernetes-dashboard-certs
 &nbsp; &nbsp; &nbsp; &nbsp;- name: tmp-volume
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;emptyDir: {}
 &nbsp; &nbsp; &nbsp;serviceAccountName: kubernetes-dashboard
 &nbsp; &nbsp; &nbsp;nodeSelector:
 &nbsp; &nbsp; &nbsp; &nbsp;&quot;kubernetes.io/os&quot;: linux
 &nbsp; &nbsp; &nbsp;# Comment the following tolerations if Dashboard must not be deployed on master
 &nbsp; &nbsp; &nbsp;tolerations:
 &nbsp; &nbsp; &nbsp; &nbsp;- key: node-role.kubernetes.io/master
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: dashboard-metrics-scraper
 &nbsp;name: dashboard-metrics-scraper
 &nbsp;namespace: kubernetes-dashboard
spec:
 &nbsp;ports:
 &nbsp; &nbsp;- port: 8000
 &nbsp; &nbsp; &nbsp;targetPort: 8000
 &nbsp;selector:
 &nbsp; &nbsp;k8s-app: dashboard-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
 &nbsp;labels:
 &nbsp; &nbsp;k8s-app: dashboard-metrics-scraper
 &nbsp;name: dashboard-metrics-scraper
 &nbsp;namespace: kubernetes-dashboard
spec:
 &nbsp;replicas: 1
 &nbsp;revisionHistoryLimit: 10
 &nbsp;selector:
 &nbsp; &nbsp;matchLabels:
 &nbsp; &nbsp; &nbsp;k8s-app: dashboard-metrics-scraper
 &nbsp;template:
 &nbsp; &nbsp;metadata:
 &nbsp; &nbsp; &nbsp;labels:
 &nbsp; &nbsp; &nbsp; &nbsp;k8s-app: dashboard-metrics-scraper
 &nbsp; &nbsp;spec:
 &nbsp; &nbsp; &nbsp;securityContext:
 &nbsp; &nbsp; &nbsp; &nbsp;seccompProfile:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;type: RuntimeDefault
 &nbsp; &nbsp; &nbsp;containers:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: dashboard-metrics-scraper
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;image: kubernetesui/metrics-scraper:v1.0.7
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ports:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- containerPort: 8000
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;protocol: TCP
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;livenessProbe:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;httpGet:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;scheme: HTTP
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;path: /
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;port: 8000
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;initialDelaySeconds: 30
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;timeoutSeconds: 30
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;volumeMounts:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- mountPath: /tmp
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;name: tmp-volume
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;securityContext:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;allowPrivilegeEscalation: false
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;readOnlyRootFilesystem: true
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;runAsUser: 1001
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;runAsGroup: 2001
 &nbsp; &nbsp; &nbsp;serviceAccountName: kubernetes-dashboard
 &nbsp; &nbsp; &nbsp;nodeSelector:
 &nbsp; &nbsp; &nbsp; &nbsp;&quot;kubernetes.io/os&quot;: linux
 &nbsp; &nbsp; &nbsp;# Comment the following tolerations if Dashboard must not be deployed on master
 &nbsp; &nbsp; &nbsp;tolerations:
 &nbsp; &nbsp; &nbsp; &nbsp;- key: node-role.kubernetes.io/master
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;effect: NoSchedule
 &nbsp; &nbsp; &nbsp;volumes:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: tmp-volume
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;emptyDir: {}
```

**11）生成dashboard自签证书**

```
$ mkdir -p /export/servers/kubernetes/certs &amp;&amp; cd /export/servers/kubernetes/certs/
$ openssl genrsa -out dashboard.key 2048
$ openssl req -days 3650 -new -key dashboard.key -out dashboard.csr -subj /C=CN/ST=BEIJING/L=BEIJING/O=JD/OU=JD/CN=172.16.16.42
$ openssl x509 -req -days 3650 -in dashboard.csr -signkey dashboard.key -out dashboard.crt
```

**12）执行以下操作命令**

```
# 去除主节点的污点
$ kubectl taint nodes --all node-role.kubernetes.io/master-
# 创建命名空间
$ kubectl create namespace kubernetes-dashboard
# 创建Secret
$ kubectl create secret tls kubernetes-dashboard-certs -n kubernetes-dashboard --key dashboard.key \
--cert dashboard.crt
```

**13）生效dashboard yml配置文件**

```
$ kubectl apply -f /export/servers/kubernetes/dashboard.yml
# 查看pods状态
$ kubectl get pods -A | grep kubernetes-dashboard
kubernetes-dashboard &nbsp; dashboard-metrics-scraper-c45b7869d-rbdt4 &nbsp; 1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 15m
kubernetes-dashboard &nbsp; kubernetes-dashboard-764b4dd7-rt66t &nbsp; &nbsp; &nbsp; &nbsp; 1/1 &nbsp; &nbsp; Running &nbsp; 0 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 15m
```

**14）访问dashboard页面**

```
# web浏览器访问地址：IP地址为集群任意节点（可以是LB地址）https://192.168.186.121:31001/#/login
```

**15）制作访问token**

```
# 新增配置文件 dashboard-adminuser.yaml
$ touch /export/servers/kubernetes/dashboard-adminuser.yaml &amp;&amp; vim /export/servers/kubernetes/dashboard-adminuser.yaml
# 输入以下内容
---
apiVersion: v1
kind: ServiceAccount
metadata:
 &nbsp;name: admin-user
 &nbsp;namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 &nbsp;name: admin-user
roleRef:
 &nbsp;apiGroup: rbac.authorization.k8s.io
 &nbsp;kind: ClusterRole
 &nbsp;name: cluster-admin
subjects:
- kind: ServiceAccount
 &nbsp;name: admin-user
 &nbsp;namespace: kubernetes-dashboard
 &nbsp;
# 执行yaml文件
$ kubectl create -f /export/servers/kubernetes/dashboard-adminuser.yaml
# 预期输出结果
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
# 说明：上面创建了一个叫admin-user的服务账号，并放在kubernetes-dashboard命名空间下，并将cluster-admin角色绑定到admin-user账户，这样admin-user账户就有了管理员的权限。默认情况下，kubeadm创建集群时已经创建了cluster-admin角色，直接绑定即可

# 查看admin-user账户的token
$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk &#39;{print $1}&#39;)
# 预期输出结果
Name: &nbsp; &nbsp; &nbsp; &nbsp; admin-user-token-9fpps
Namespace: &nbsp; &nbsp;kubernetes-dashboard
Labels:<none>Annotations: &nbsp;kubernetes.io/service-account.name: admin-user
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;kubernetes.io/service-account.uid: 72c1aa28-6385-4d1a-b22c-42427b74b4c7

Type: &nbsp;kubernetes.io/service-account-token

Data
====
ca.crt: &nbsp; &nbsp; 1099 bytes
namespace: &nbsp;20 bytes
token: &nbsp; &nbsp; &nbsp;eyJhbGciOiJSUzI1NiIsImtpZCI6IjFEckU0NXB5Yno5UV9MUFkxSUpPenJhcTFuektHazM1c2QzTGFmRzNES0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTlmcHBzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3MmMxYWEyOC02Mzg1LTRkMWEtYjIyYy00MjQyN2I3NGI0YzciLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.oA3NLhhTaXd2qvWrPDXat2w9ywdWi_77SINk4vWkfIIzMmxBEHnqvDIBvhRC3frIokNSvT71y6mXN0KHu32hBa1YWi0MuzF165ZNFtM_rSQiq9OnPxeFvLaKS-0Vzr2nWuBx_-fTt7gESReSMLEJStbPb1wOnR6kqtY66ajKK5ILeIQ77I0KXYIi7GlPEyc6q4bIjweZ0HSXDPR4JSnEAhrP8Qslrv3Oft4QZVNj47x7xKC4dyyZOMHUIj9QhkpI2gMbiZ8XDUmNok070yDc0TCxeTZKDuvdsigxCMQx6AesD-8dca5Hb8Sm4mEPkGJekvMzkLkM97y_pOBPkfTAIA

# 把上面命令执行获取到的Token复制到登录界面的Token输入框中，即可正常登录dashboard
```

**13）登录dashboard如下**

![](https://s3.cn-north-1.jdcloud-oss.com/shendengbucket1/2022-10-24-07-26MtFtfpXLrPdBCQG.png)

### kubectl安装

**1\. 环境说明**

| **名称** | **说明** |
| --- | --- |
| **操作系统** | CentOS Linux release 7.8.2003 |
| **kubectl** | kubectl-1.22.4-0.x86_64 |
| **节点** | deploy |

**2\. 部署说明**

Kubernetes kubectl客户端。

**3\. 解压之前上传的kubadm-rpm包**

```
$ tar xzvf kubeadm-rpm.tgz 
```

**4\. 执行安装**

```
$ rpm -ivh bc7a9f8e7c6844cfeab2066a84b8fecf8cf608581e56f6f96f80211250f9a5e7-kubectl-1.22.4-0.x86_64.rpm
```

**5\. 增加执行权限**

```
# 生成kubelet环境配置文件
$ mkdir -p $HOME/.kube
$ sudo touch $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 从任意master节点复制内容至上面的配置文件
```

**6\. 查看版本**

```
$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;22&quot;, GitVersion:&quot;v1.22.4&quot;, GitCommit:&quot;b695d79d4f967c403a96986f1750a35eb75e75f1&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-11-17T15:48:33Z&quot;, GoVersion:&quot;go1.16.10&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;22&quot;, GitVersion:&quot;v1.22.4&quot;, GitCommit:&quot;b695d79d4f967c403a96986f1750a35eb75e75f1&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-11-17T15:42:41Z&quot;, GoVersion:&quot;go1.16.10&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
```

## helm安装

**1\. 环境说明**

| **名称** | **说明** |
| --- | --- |
| **操作系统** | CentOS Linux release 7.8.2003 |
| **helm** | helm-v3.9.3-linux-amd64.tar.gz |
| **节点** | deploy |

**2\. 部署说明**

Kubernetes资源包及配置管理工具。

**3\. 下载helm离线安装包并上传至服务器**

```
$ wget https://get.helm.sh/helm-v3.9.3-linux-amd64.tar.gz
```

**4\. 解压安装包**

```
$ tar -zxvf helm-v3.9.3-linux-amd64.tar.gz -C /export/servers/
$ cd /export/servers/linux-amd64
```

**5\. 增加执行权限**

```
$ cp linux-amd64/helm /usr/local/bin/
$ chmod +x /usr/local/bin/helm
```

**6\. 查看版本**

```
$ helm version
version.BuildInfo{Version:&quot;v3.9.3&quot;, GitCommit:&quot;414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58&quot;, GitTreeState:&quot;clean&quot;, GoVersion:&quot;go1.17.13&quot;}
```

**设置本地存储挂载nas**

```
$ mkdir /export/servers/helm_chart/local-path-storage &amp;&amp; cd /export/servers/helm_chart/local-path-storage/local-path-storage.yaml
$ vim local-path-storage.yaml
# 设置以下内容，设置&quot;paths&quot;:[&quot;/home/admin/local-path-provisioner&quot;] 为nas目录，没有目录需要创建
apiVersion: v1
kind: Namespace
metadata:
 &nbsp;name: local-path-storage
 
---
apiVersion: v1
kind: ServiceAccount
metadata:
 &nbsp;name: local-path-provisioner-service-account
 &nbsp;namespace: local-path-storage
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 &nbsp;name: local-path-provisioner-role
rules:
 &nbsp;- apiGroups: [ &quot;&quot; ]
 &nbsp; &nbsp;resources: [ &quot;nodes&quot;, &quot;persistentvolumeclaims&quot;, &quot;configmaps&quot; ]
 &nbsp; &nbsp;verbs: [ &quot;get&quot;, &quot;list&quot;, &quot;watch&quot; ]
 &nbsp;- apiGroups: [ &quot;&quot; ]
 &nbsp; &nbsp;resources: [ &quot;endpoints&quot;, &quot;persistentvolumes&quot;, &quot;pods&quot; ]
 &nbsp; &nbsp;verbs: [ &quot;*&quot; ]
 &nbsp;- apiGroups: [ &quot;&quot; ]
 &nbsp; &nbsp;resources: [ &quot;events&quot; ]
 &nbsp; &nbsp;verbs: [ &quot;create&quot;, &quot;patch&quot; ]
 &nbsp;- apiGroups: [ &quot;storage.k8s.io&quot; ]
 &nbsp; &nbsp;resources: [ &quot;storageclasses&quot; ]
 &nbsp; &nbsp;verbs: [ &quot;get&quot;, &quot;list&quot;, &quot;watch&quot; ]
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 &nbsp;name: local-path-provisioner-bind
roleRef:
 &nbsp;apiGroup: rbac.authorization.k8s.io
 &nbsp;kind: ClusterRole
 &nbsp;name: local-path-provisioner-role
subjects:
 &nbsp;- kind: ServiceAccount
 &nbsp; &nbsp;name: local-path-provisioner-service-account
 &nbsp; &nbsp;namespace: local-path-storage
 
---
apiVersion: apps/v1
kind: Deployment
metadata:
 &nbsp;name: local-path-provisioner
 &nbsp;namespace: local-path-storage
spec:
 &nbsp;replicas: 1
 &nbsp;selector:
 &nbsp; &nbsp;matchLabels:
 &nbsp; &nbsp; &nbsp;app: local-path-provisioner
 &nbsp;template:
 &nbsp; &nbsp;metadata:
 &nbsp; &nbsp; &nbsp;labels:
 &nbsp; &nbsp; &nbsp; &nbsp;app: local-path-provisioner
 &nbsp; &nbsp;spec:
 &nbsp; &nbsp; &nbsp;serviceAccountName: local-path-provisioner-service-account
 &nbsp; &nbsp; &nbsp;containers:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: local-path-provisioner
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;image: rancher/local-path-provisioner:v0.0.21
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;imagePullPolicy: IfNotPresent
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;command:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- local-path-provisioner
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- --debug
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- start
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- --config
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- /etc/config/config.json
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;volumeMounts:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- name: config-volume
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;mountPath: /etc/config/
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;env:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- name: POD_NAMESPACE
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;valueFrom:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;fieldRef:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;fieldPath: metadata.namespace
 &nbsp; &nbsp; &nbsp;volumes:
 &nbsp; &nbsp; &nbsp; &nbsp;- name: config-volume
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;configMap:
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;name: local-path-config
 
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 &nbsp;name: local-path
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
 
---
kind: ConfigMap
apiVersion: v1
metadata:
 &nbsp;name: local-path-config
 &nbsp;namespace: local-path-storage
data:
 &nbsp;config.json: |-
 &nbsp; &nbsp;{
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;nodePathMap&quot;:[
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;node&quot;:&quot;DEFAULT_PATH_FOR_NON_LISTED_NODES&quot;,
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&quot;paths&quot;:[&quot;/nas_data/jdiot/local-path-provisioner&quot;] 
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;}
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;]
 &nbsp; &nbsp;}
 &nbsp;setup: |-
 &nbsp; &nbsp;#!/bin/sh
 &nbsp; &nbsp;while getopts &quot;m:s:p:&quot; opt
 &nbsp; &nbsp;do
 &nbsp; &nbsp; &nbsp; &nbsp;case $opt in
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;p)
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;absolutePath=$OPTARG
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;;;
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;s)
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;sizeInBytes=$OPTARG
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;;;
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;m)
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;volMode=$OPTARG
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;;;
 &nbsp; &nbsp; &nbsp; &nbsp;esac
 &nbsp; &nbsp;done
 
 &nbsp; &nbsp;mkdir -m 0777 -p ${absolutePath}
 &nbsp;teardown: |-
 &nbsp; &nbsp;#!/bin/sh
 &nbsp; &nbsp;while getopts &quot;m:s:p:&quot; opt
 &nbsp; &nbsp;do
 &nbsp; &nbsp; &nbsp; &nbsp;case $opt in
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;p)
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;absolutePath=$OPTARG
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;;;
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;s)
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;sizeInBytes=$OPTARG
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;;;
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;m)
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;volMode=$OPTARG
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;;;
 &nbsp; &nbsp; &nbsp; &nbsp;esac
 &nbsp; &nbsp;done
 
 &nbsp; &nbsp;rm -rf ${absolutePath}
 &nbsp;helperPod.yaml: |-
 &nbsp; &nbsp;apiVersion: v1
 &nbsp; &nbsp;kind: Pod
 &nbsp; &nbsp;metadata:
 &nbsp; &nbsp; &nbsp;name: helper-pod
 &nbsp; &nbsp;spec:
 &nbsp; &nbsp; &nbsp;containers:
 &nbsp; &nbsp; &nbsp;- name: helper-pod
 &nbsp; &nbsp; &nbsp; &nbsp;image: busybox
```

注：以上依赖镜像需要从公网环境下载依赖并导入镜像库，需要设置以上对应镜像地址从私有镜像库拉取镜像

**生效本地存储yaml**

```
$ kubectl apply -f local-path-storage.yaml -n local-path-storage
```

**设置k8s默认存储**

```
$ kubectl patch storageclass local-path &nbsp;-p &#39;{&quot;metadata&quot;: {&quot;annotations&quot;:{&quot;storageclass.kubernetes.io/is-default-class&quot;:&quot;true&quot;}}}&#39;
```

注：后面部署的中间件及服务需要修改对应的存储为本地存储：&quot;storageClass&quot;: &quot;local-path&quot;

#### 作者：郝建伟</none></none></none></none>