分享程序员开发的那些事...
更新时间：2022-08-14 21:51:53
" 本文重点讲解从无到有来搭建一套生产环境可用的K8S高可用集群，搭建完K8S集群之后，再来部署Harbor镜像仓库为容器镜像提供存储服务，然后在K8S集群中部署Ingress-nginx组件用于发布集群中的应用程序，最后再使用Helm在K8S集群中搭建一套高可用集群模式的Rancher可视化仪表盘，用于管理K8S集群。"
????个人主页????：jiangxl~
????个人简介????：华为云享专家、51CTO专家博主、DevOps运维领域优质创作者、 2021年度博客之星运维与安全领域TOP1。
????博客领域????：云原生、云计算、DevOps运维开发。
⭐️写作风格⭐️：每篇文章详细程度满分十分可用给到九分，坚持图文并茂、有图有文字的来讲解原理架构。
⭐️推荐学习专栏⭐️：
@[toc]
sealos旨在做一个简单干净轻量级稳定的kubernetes安装工具，能很好的支持高可用安装。
sealos一条命令就可以实现k8s高可用。
sealos特性与优势：
sealos clean --all清楚集群时，只有sealos安装的组件才会自动删除，否则将会存留在系统。
环境准备：
| IP | 主机名 |
|---|---|
| 192.168.16.106 | k8s-master1 |
| 192.168.16.105 | k8s-master2 |
| 192.168.16.107 | k8s-node1 |
| 192.168.16.104 | k8s-node2 |
1.设置主机名
hostnamectl set-hostname k8s-master1
hostnamectl set-hostname k8s-master2
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2
2.关闭selinux及防火墙
setenforce 0
sed -ri '/^SELINUX=/c SELINUX=disabled' /etc/sysconfig/selinux
sed -ri '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld1.sealos官网获取软件包
[root@k8s-master1 ~/soft]# ll
总用量 507472
-rw-r--r-- 1 root root 475325715 4月 27 15:21 kube1.19.6.tar.gz
-rw-r--r-- 1 root root 44322816 4月 29 13:47 sealos
2.将kube1.19.6.tar.gz复制到所有k8s节点
[root@k8s-master1 ~/soft]# for i in {4,5,7}
do
scp -r /root/soft root@192.168.16.10${i}:/root
done
3.安装sealos
[root@k8s-master1 ~/soft]# chmod +x sealos && mv sealos /usr/bin
[root@k8s-master1 ~/soft]# sealos version
Version: 3.3.9-rc.3
Last Commit: 4db4953
Build Date: 2021-04-10T11:25:04Z[root@k8s-master1 ~]# sealos init --passwd 'teacher' --master 192.168.16.106 --master 192.168.16.105 --node 192.168.16.104 --node 192.168.16.107 --pkg-url /root/soft/kube1.19.6.tar.gz --version v1.19.6[root@k8s-master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready master 3m50s v1.19.6
k8s-master2 Ready master 3m11s v1.19.6
k8s-node1 Ready <none> 2m17s v1.19.6
k8s-node2 Ready <none> 2m19s v1.19.6
#详细信息
[root@k8s-master1 ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master1 Ready master 2m55s v1.19.6 192.168.16.106 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://19.3.12
k8s-master2 Ready master 2m18s v1.19.6 192.168.16.105 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://19.3.12
k8s-node1 Ready <none> 83s v1.19.6 192.168.16.104 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://19.3.12
k8s-node2 Ready <none> 85s v1.19.6 192.168.16.107 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://19.3.12sealos init --passwd '123456' \
--master 192.168.0.2 --master 192.168.0.3 --master 192.168.0.4 \
--node 192.168.0.5 \
--pkg-url /root/kube1.20.0.tar.gz \
--version v1.20.0#增加一个master节点
sealos join --master 192.168.0.6
#增加多个master节点
sealos join --master 192.168.0.6 --master 192.168.0.7
#增加多个master节点，IP是连续的
sealos join --master 192.168.0.6-192.168.0.9 #或者多个连续IP#增加一个master节点
sealos join --node 192.168.0.6
#增加多个master节点
sealos join --node 192.168.0.6 --node 192.168.0.7
#增加多个master节点，IP是连续的
sealos join --node 192.168.0.6-192.168.0.9 #或者多个连续IP#删除一个master节点
sealos clean --master 192.168.0.6
#删除多个master节点
sealos clean --master 192.168.0.6 --master 192.168.0.7
#删除多个master节点，IP是连续的
sealos clean --master 192.168.0.6-192.168.0.9 #或者多个连续IP#删除一个master节点
sealos clean --node 192.168.0.6
#删除多个master节点
sealos clean --node 192.168.0.6 --node 192.168.0.7
#删除多个master节点，IP是连续的
sealos clean --node 192.168.0.6-192.168.0.9 #或者多个连续IPsealos clean --allsealos etcd save1.下载harbor
https://github.com/goharbor/harbor/releases/download
2.解压harbor
[root@k8s-master1 ~]# tar xf harbor-offline-installer-v1.6.1.tgz
[root@k8s-master1 ~]# cd harbor/
3.配置harbor
[root@k8s-master1 ~/harbor]# vim harbor.cfg
hostname = harbor.jiangxl.com
harbor_admin_password = admin
4.部署harbor
[root@k8s-master1 ~/harbor]# ./install.sh访问http://harbor.jiangxl.com/harbor/projects账号密码admin
1.获取ingress-nginx yaml文件
[root@k8s-master1 ~/k8s1.19/ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
[root@k8s-master1 ~/k8s1.19/ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/baremetal/service-nodeport.yaml
2.调整mandatory.yaml中的镜像地址为国内镜像
[root@k8s-master1 ~/k8s1.19/ingress]# vim mandatory.yaml
image: registry.cn-beijing.aliyuncs.com/google_registry/nginx-ingress-controller:0.30.0
#在220行左右
3.调整ingress svc暴露端口，映射主机的443和80端口
[root@k8s-master1 ~/ingress]# vim service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
nodePort: 80
- name: https
port: 443
targetPort: 443
protocol: TCP
nodePort: 443
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
3.创建资源
[root@k8s-master ~/k8s1.19/ingress]# kubectl apply -f ./
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
limitrange/ingress-nginx created
service/ingress-nginx created
4.查看资源状态
[root@k8s-master1 ~/ingress]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/nginx-ingress-controller-766867958b-vp7fc 1/1 Running 0 36s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx NodePort 10.111.10.249 <none> 80:80/TCP,443:443/TCP 36s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-ingress-controller 1/1 1 1 36s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-ingress-controller-766867958b 1 1 1 36s使用helm部署rancher高可用集群，并通过ingress暴露rancher本身就是ingress暴露，无需再创建ingress资源，只需要准备好ingress所需要的的证书文件就可以了
1.下载helm
https://sourceforge.net/projects/helm.mirror/files/latest/download
2.解压使用helm
[root@k8s-master ~/soft]# tar xf helm.tar.gz
[root@k8s-master ~/soft]# cp helm /usr/bin/
[root@k8s-master ~/soft]# helm
3.上传TLS证书生成软件
[root@k8s-master ~/soft]# tar xf TLS.tar.gz -C /root/
[root@k8s-master ~]# cd TLS/
[root@k8s-master1 ~/TLS]#
总用量 18852
-rw-r--r-- 1 root root 294 3月 10 15:11 ca-config.json
-rw-r--r-- 1 root root 960 3月 10 15:11 ca.csr
-rw-r--r-- 1 root root 212 3月 10 15:11 ca-csr.json
-rw------- 1 root root 1675 3月 10 15:11 ca-key.pem
-rw-r--r-- 1 root root 1273 3月 10 15:11 ca.pem
-rwxr-xr-x 1 root root 999 3月 10 15:11 certs.sh
-rwxr-xr-x 1 root root 10376657 10月 2 2019 cfssl
-rwxr-xr-x 1 root root 6595195 10月 2 2019 cfssl-certinfo
-rwxr-xr-x 1 root root 2277873 10月 2 2019 cfssljson
-rwxr-xr-x 1 root root 344 10月 3 2019 cfssl.sh
-rw-r--r-- 1 root root 195 3月 10 15:11 rancher-csr.json
-rw-r--r-- 1 root root 976 3月 10 15:11 rancher.teacher.com.cn.csr
-rw------- 1 root root 1675 3月 10 15:11 rancher.teacher.com.cn-key.pem
-rw-r--r-- 1 root root 1326 3月 10 15:11 rancher.teacher.com.cn.pem
4.删除不需要的证书文件
#删除的文件都是上次使用时生成的
[root@k8s-master ~/soft/TLS]# rm -rf ca*
[root@k8s-master ~/soft/TLS]# rm -rf rancher*
[root@k8s-master ~/soft/TLS]# ll
总用量 18816
-rwxr-xr-x 1 root root 999 3月 10 15:11 certs.sh
-rwxr-xr-x 1 root root 10376657 10月 2 2019 cfssl
-rwxr-xr-x 1 root root 6595195 10月 2 2019 cfssl-certinfo
-rwxr-xr-x 1 root root 2277873 10月 2 2019 cfssljson
-rwxr-xr-x 1 root root 344 10月 3 2019 cfssl.sh
1.准备生成证书所需要的的命令
[root@k8s-master ~/TLS]# cat cfssl.sh
#curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
#curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
#curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
cp -rf cfssl cfssl-certinfo cfssljson /usr/local/bin
chmod +x /usr/local/bin/cfssl*
[root@k8s-master ~/soft/TLS]# ./cfssl.sh
2.编辑证书所使用的域名
[root@k8s-master ~/soft/TLS]# vim certs.sh
cat > rancher-csr.json <<EOF
{
"CN": "rancher.jiangxl.com", #改成自己的域名
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes rancher-csr.json | cfssljson -bare rancher.jiangxl.com #改成自己的域名
[root@k8s-master ~/soft/TLS]# ./certs.sh
2021/05/07 15:08:05 [INFO] generating a new CA key and certificate from CSR
2021/05/07 15:08:05 [INFO] generate received request
2021/05/07 15:08:05 [INFO] received CSR
2021/05/07 15:08:05 [INFO] generating key: rsa-2048
2021/05/07 15:08:05 [INFO] encoded CSR
2021/05/07 15:08:05 [INFO] signed certificate with serial number 101654790097278429805640175100193242973661583912
2021/05/07 15:08:05 [INFO] generate received request
2021/05/07 15:08:05 [INFO] received CSR
2021/05/07 15:08:05 [INFO] generating key: rsa-2048
2021/05/07 15:08:06 [INFO] encoded CSR
2021/05/07 15:08:06 [INFO] signed certificate with serial number 256114560036375503495441829622686649009249955560
2021/05/07 15:08:06 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").[root@k8s-master ~]# kubectl create ns cattle-system
namespace/cattle-system created[root@k8s-master ~]# helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
"rancher-latest" has been added to your repositories
[root@k8s-master ~]# helm repo ls
NAME URL
rancher-latest https://releases.rancher.com/server-charts/latest如果使用ingress暴露rancher则执行下面的命令创建证书
1.生成ingress所需要使用的证书secret
[root@k8s-master1 ~/TLS]# kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=rancher.jiangxl.com.pem --key=rancher.jiangxl.com-key.pem
secret/tls-rancher-ingress created
2.创建rancher所需要的的证书
[root@k8s-master1 ~/TLS]# cp /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/cacerts.pem
[root@k8s-master1 ~/TLS]# kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem=/etc/kubernetes/pki/cacerts.pem
secret/tls-ca created
1.创建rancher
[root@k8s-master1 ~/TLS]# helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=rancher.jiangxl.com \
--set ingress.tls.source=tls-rancher-ingress \ #填写6.5中创建ingress secret
--set privateCA=true
NAME: rancher
LAST DEPLOYED: Sat May 8 14:31:27 2021
NAMESPACE: cattle-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Rancher Server has been installed.
NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued and Ingress comes up.
Check out our docs at https://rancher.com/docs/rancher/v2.x/en/
Browse to https://rancher.jiangxl.com
Happy Containering!
2.查看创建的rancher
[root@k8s-master1 ~/TLS]# helm ls -n cattle-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
rancher cattle-system 1 2021-05-08 14:31:27.494706017 +0800 CST deployed rancher-2.5.8 v2.5.8 [root@k8s-master1 ~/TLS]# kubectl get all -n cattle-system
NAME READY STATUS RESTARTS AGE
pod/rancher-5bd5d98557-k5cd7 1/1 Running 0 38s
pod/rancher-5bd5d98557-pmhzh 1/1 Running 0 38s
pod/rancher-5bd5d98557-vgtwq 1/1 Running 0 38s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/rancher ClusterIP 10.108.97.195 <none> 80/TCP,443/TCP 38s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/rancher 3/3 3 3 38s
NAME DESIRED CURRENT READY AGE
replicaset.apps/rancher-5bd5d98557 3 3 3 38s
[root@k8s-master1 ~/ingress]# kubectl get pod,deploy,svc,ingress,secret -n cattle-system
NAME READY STATUS RESTARTS AGE
pod/rancher-5bd5d98557-5tphq 1/1 Running 0 11m
pod/rancher-5bd5d98557-bjcjh 1/1 Running 0 11m
pod/rancher-5bd5d98557-rhkpp 1/1 Running 0 11m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/rancher 3/3 3 3 11m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/rancher ClusterIP 10.111.131.250 <none> 80/TCP,443/TCP 11m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.extensions/rancher <none> rancher.jiangxl.com 10.111.10.249 80, 443 11m
NAME TYPE DATA AGE
secret/default-token-clf8n kubernetes.io/service-account-token 3 12m
secret/rancher-token-lhgn2 kubernetes.io/service-account-token 3 11m
secret/serving-cert kubernetes.io/tls 2 61s
secret/sh.helm.release.v1.rancher.v1 helm.sh/release.v1 1 11m
secret/tls-ca Opaque 1 2m39s
secret/tls-rancher kubernetes.io/tls 2 61s
secret/tls-rancher-ingress kubernetes.io/tls 2 12m
secret/tls-rancher-internal-ca kubernetes.io/tls 2 61s
我们是通过ingress绑定域名暴露的rancher，因此要通过https://集群任意节点ip:ingress-https端口进行访问。
1）查出ingress https的端口
[root@k8s-master1 ~/k8s1.19/ingress]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx NodePort 10.99.98.227 <none> 80:80/TCP,443:443/TCP 166m
1）设置密码
3）设置URL
3）登陆rancher
4）查看集群
k8s-master2原来装个k8s1.15版本，由于种种原因，再次使用sealos安装高版本的k8s时就会报错，错误内容如下：
[root@k8s-master2 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: inactive (dead)
Docs: http://kubernetes.io/docs/
5月 06 17:18:25 k8s-master2 kubelet[16231]: created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
5月 06 17:18:25 k8s-master2 kubelet[16231]: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/...o:58 +0x8a
5月 06 17:18:25 k8s-master2 kubelet[16231]: goroutine 60 [select]:
5月 06 17:18:25 k8s-master2 kubelet[16231]: k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.(*worker).start(0xc00082a7d0)
5月 06 17:18:25 k8s-master2 kubelet[16231]: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view...154 +0x105
5月 06 17:18:25 k8s-master2 systemd[1]: Unit kubelet.service entered failed state.
5月 06 17:18:25 k8s-master2 kubelet[16231]: created by k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.init.0
5月 06 17:18:25 k8s-master2 kubelet[16231]: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view...o:32 +0x57
5月 06 17:18:25 k8s-master2 systemd[1]: kubelet.service failed.
5月 06 17:18:34 k8s-master2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Hint: Some lines were ellipsized, use -l to show in full.
排查思路： 首先查看下kubelet的日志，日志记录的东西都是最详细的。
[root@k8s-master2 ~]# journalctl -xeu kubelet | less经过仔细排查发现少了一个bootstrap-kubelet.conf的文件，全新的机器安装k8s集群是不需要bootstrap-kubelet.conf这个文件的，会产生这种报错都是因为这个机器之前装个k8s。
解决方法： bootstrap-kubelet.conf这个文件就是/etc/kubernetes/admin.conf文件，只需要cp一下即可。
[root@k8s-master2 ~]# cd /etc/kubernetes/
[root@k8s-master2 /etc/kubernetes]# cp admin.conf bootstrap-kubelet.conf
[root@k8s-master2 /etc/kubernetes]# systemctl restart kubelet重启后问题就解决了。
如果该机器彻底删除了docker和k8s，依旧使用sealos部署k8s报错如下，那么可能就是docker没有安装成功，可以检查下docker的安装包全不全和docker服务有没有启动，docker都安装失败，kubelet一定会报错，因为kubeadm安装的k8s，所有组件都是以pod形式运行。
看如下报错应该是docker的问题，docker没有启动，但是一查发现没有安装docker，但是系统原来存在过docker，sealos检测到有docker命令就不会安装docker，因此需要把docker完全卸载掉，重新安装集群即可。
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
rm -rf /etc/systemd/system/docker.service.d
rm -rf /var/lib/docker
rm -rf /var/run/docker
umount /run/docker/netns/default
rm -rf /run/docker/netns/default
rm -rf /usr/bin/docker*
kubeadm reset -f
modprobe -r ipip
lsmod
rm -rf ~/.kube/
rm -rf /etc/kubernetes/
rm -rf /etc/systemd/system/kubelet.service.d
rm -rf /etc/systemd/system/kubelet.service
rm -rf /usr/bin/kube*
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/etcd
rm -rf /var/etcd