广告

Kubernetes集群部署教程二·Master节点

Kubernetes(k8s)是Google开源的一个容器编排引擎,它支持自动化部署、大规模可伸缩、应用容器化管理,具备可移植、可扩展、自动化等特点。

本系列教程讲述在CentOS7系统中k8s集群的部署过程,本节内容重点讲述如何部署master节点。

核心组件介绍

名称 介绍
kube-apiserver Kubernetes API,集群的统一稿,各组件协调者,以 HTTP API 提供接口服务,所有对象资源的增删改查和监听操作都交给APIServer处理后再提交给etcd存储。
kube-controller-manager 处理集群中常规后台任务,一个资源对应一个控制器,而ControllerManager就是负责管理这些控制器的。
kube-scheduler 根据调度算法为新创建的Pod选择一个Node节点。

配置Mater环境

0x01 安装应用包

在Master上安装以下应用包:

# yum -y install ebtables ethtool

yum -y install docker-ce kubelet kubeadm kubectl

systemctl enable docker kubelet

systemctl start docker

0x02 检查cgroup

以下三个方法中任选一个,不管用systemd还是cgroupfs,统一即可,建议方法2

方法1:在/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf加入参数

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

方法2:在/usr/lib/systemd/system/docker.service的ExecStart=/usr/bin/dockerd行最后添加参数

--exec-opt native.cgroupdriver=systemd

方法3:编辑/etc/docker/daemon.json

"exec-opts": ["native.cgroupdriver=systemd"]

统一cgroup后重启服务:

systemctl daemon-reload

systemctl restart docker

配置Master节点

1x01 配置高可用

安装haproxy和keepalived:

yum install -y haproxy keepalived

配置haproxy:

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global

    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode    http
    log     global
    retries 1
    timeout http-request 10s
    timeout queue 20s
    timeout connect 5s
    timeout client 20s
    timeout server 20s
    timeout http-keep-alive 10s
    timeout check 10s

listen admin_stats
    mode  http
    bind  0.0.0.0:1080
    log   127.0.0.1 local0 err
    stats refresh 30s
    stats uri /haproxy-status
    stats realm Haproxy\ Statistics
    stats auth admin:admin
    stats hide-version
    stats admin if TRUE

#---------------------------------------------------------------------
# apiserver frontend which proxys to the masters
#---------------------------------------------------------------------
frontend apiserver
    bind            *:8443
    mode            tcp
    option          tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option     httpchk GET /healthz
    http-check expect status 200
    mode       tcp
    option     ssl-hello-chk
    balance    roundrobin
    server     k8sm01 10.10.200.201:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
    server     k8sm02 10.10.200.202:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
    server     k8sm03 10.10.200.203:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3

编辑守护脚本:

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:8443/ -o /dev/null || errorExit "Error GET https://localhost:8443/"
if ip addr | grep -q 10.10.200.200; then
    curl --silent --max-time 2 --insecure https://10.10.200.200:8443/ -o /dev/null || errorExit "Error GET https://10.26.25.23:8443/"
fi

chmod +x /etc/keepalived/check_apiserver.sh

配置keepalived(MASTER):

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    router_id LVS_K8SM
}

vrrp_script check_apiserver {
    script   "/etc/keepalived/check_apiserver.sh"
    interval 3
    weight  -2
    fall     10
    rise     2
}

vrrp_instance VI_1 {
    state             MASTER
    interface         ens33
    virtual_router_id 51
    priority          100
    authentication {
        auth_type     PASS
        auth_pass     kubernetes
    }
    virtual_ipaddress {
        10.10.200.200
    }
    track_script {
        check_apiserver
    }
}

配置keepalived(BACKUP):

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    router_id LVS_K8SM
}

vrrp_script check_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 3
    weight  -2
    fall     10
    rise     2
}

vrrp_instance VI_1 {
    state             BACKUP
    interface         ens33
    virtual_router_id 51
    priority          90
    authentication {
        auth_type     PASS
        auth_pass     kubernetes
    }
    virtual_ipaddress {
        10.10.200.200
    }
    track_script {
        check_apiserver
    }
}

启动haproxy和keepalived:

systemctl start keepalived haproxy
systemctl enable keepalived haproxy

1x02 导入配置镜像文件

查看“kubeadm config images list”命令需要拉取的镜像:

  • k8s.gcr.io/kube-apiserver:v1.19.2
  • k8s.gcr.io/kube-controller-manager:v1.19.2
  • k8s.gcr.io/kube-scheduler:v1.19.2
  • k8s.gcr.io/kube-proxy:v1.19.2
  • k8s.gcr.io/pause:3.2
  • k8s.gcr.io/etcd:3.4.13-0
  • k8s.gcr.io/coredns:1.7.0

# 开始拉取镜像

# kubeadm config images pull

因为某些原因,国内无法访问国外的某些资源,这里我们改用阿里云的镜像服务器,使用以下脚本拉取镜像并贴上标签:

#!/bin/bash

images=(kube-apiserver:v1.19.2 kube-controller-manager:v1.19.2 kube-scheduler:v1.19.2 kube-proxy:v1.19.2 pause:3.2 etcd:3.4.13-0 coredns:1.7.0)

for image in ${images[@]}
do
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
done

1x03 初始化master节点

切记!仅首master节点需要初始化,备用master节点跳过此步骤

kubeadm init \
--pod-network-cidr=10.232.0.0/16 \
--service-cidr=10.196.0.0/16
--control-plane-endpoint=10.10.200.200:8443

如果初始化顺利,则会显示如下日志信息,提示如何添加master/worker节点:

...

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

...
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 10.10.200.200:8443 --token bf1bni.w11z5kymeaomqddr \
    --discovery-token-ca-cert-hash sha256:26e4de3f6e72ed2b143627cf10068c84e094c2fa30c0aa2fee32f53a2b0254f2 \
    --control-plane 

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.200.200:8443 --token bf1bni.w11z5kymeaomqddr \
    --discovery-token-ca-cert-hash sha256:26e4de3f6e72ed2b143627cf10068c84e094c2fa30c0aa2fee32f53a2b0254f2 

如果 kubeadm init ... 执行失败,必须通过以下命令重置,然后重新操作:

kubeadm reset

附:kubeadm ini 参数说明

参数 说明
--apiserver-advertise-address API Server 将要广播的监听地址
--apiserver-bind-port API Server 绑定的端口,默认“6443”
--apiserver-cert-extra-sans 可选的额外提供的证书主题别名(SANs)用于指定API Server的服务器证书
--cert-dir 证书的存储路径,默认“/etc/kubernetes/pki”
--config kubeadm配置文件的路径
--cri-socket 要连接的 CRI socket 文件,默认“/var/run/dockershim.sock”
--dry-run 只输出将要执行的操作,不应用任何改变
--feature-gates 键值对的集合,用来控制各种功能的开关,默认“Auditing=false, CoreDNS=true, DynamicKubeletConfig=false”
-h, --help 获取init命令的帮助信息
--ignore-preflight-errors 忽视检查项错误列表,列表中的每一个检查项如发生错误将被展示输出为警告,而非错误
--kubernetes-version 为 control plane 选择一个特定的Kubernetes版本,默认“stable-1”
--node-name 指定节点的名称
--pod-network-cidr 指明pod网络可以使用的IP地址段
--service-cidr 为service的虚拟IP地址另外指定IP地址段,默认“10.96.0.0/12”
--service-dns-domain 为services另外指定域名,默认“cluster.local”
--skip-token-print 不打印出由 `kubeadm init` 命令生成的默认令牌
--token 这个令牌用于建立主从节点间的双向受信链接
--token-ttl 令牌被自动删除前的可用时长,设置为 '0', 令牌将永不过期,默认“24h0m0s”

1x04 配置kubectl

备用master节点跳过此步骤。

mkdir ~/.kube

cp -i /etc/kubernetes/admin.conf ~/.kube/config

# sudo chown $(id -u):$(id -g) ~/.kube/config

echo "export KUBECONFIG=~/.kube/config" >> ~/.bash_profile

source ~/.bash_profile

1x05 配置网络

备用master节点加入集群前无法操作。

方案1:采用flannel (推荐)

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

方案2:采用calico

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

1x06 验证master

备用master节点加入集群前无法操作。

kubectl get componentstatus

此时发现controller-manager和scheduler的状态是“Unhealthy”,修改它们的配置文件,去掉配置中的“--port=0”,然后重启kubelet:

sed -i '/port=0/d' /etc/kubernetes/manifests/kube-scheduler.yaml

sed -i '/port=0/d' /etc/kubernetes/manifests/kube-controller-manager.yaml

systemctl restart kubelet

继续检验master健康状态:

kubectl get nodes

kubectl get pod --all-namespaces

使Master参与Pod调度

备用master节点加入集群前无法操作。

# 参与POD负载

kubectl taint nodes --all node-role.kubernetes.io/master-

# 不参与POD负载

kubectl taint nodes <node-name> node-role.kubernetes.io/master=:NoSchedule

# 不参与POD负载并驱逐Node上已经存在的Pod

kubectl taint nodes <node-name> node-role.kubernetes.io/master=:NoExecute

配置Master集群

2x01 在首master节点操作

在首master节点验证证书有效时间(notBefore为生效时间/notAfter为失效时间):

for crt in $(find /etc/kubernetes/pki/ -name "*.crt"); do openssl x509 -in $crt -noout -dates; done

先做免密:

ssh-keygen -t rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

for n in 2 3; do ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8sm0$n; done

编辑脚本,将master证书拷贝到备用节点:

USER=root
BAK_MASTERS="10.10.200.202 10.10.200.203"
for host in ${BAK_MASTERS}
do
  scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:
  scp /etc/kubernetes/pki/ca.key "${USER}"@$host:
  scp /etc/kubernetes/pki/sa.key "${USER}"@$host:
  scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:
  scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:
  scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:
  scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt
  scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key
  scp /etc/kubernetes/admin.conf "${USER}"@$host:
done

2x02 在备用master节点操作

创建证书目录:

mkdir -p /etc/kubernetes/pki/etcd

mkdir ~/.kube

依次将首master节点复制过来的证书移至对应的目录中,并使配置生效:

cd ~/

mv admin.conf ~/.kube/config

mv etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt

mv etcd-ca.key /etc/kubernetes/pki/etcd/ca.key

mv ca.* front-proxy-ca.* sa.* /etc/kubernetes/pki/

echo "export KUBECONFIG=~/.kube/config" >> ~/.bash_profile

source ~/.bash_profile

然后执行以下命令加入集群:

kubeadm join 10.10.200.200:8443 --token bf1bni.w11z5kymeaomqddr \
--discovery-token-ca-cert-hash sha256:26e4de3f6e72ed2b143627cf10068c84e094c2fa30c0aa2fee32f53a2b0254f2 \
--control-plane

赞 (0) 打赏

精彩点评 0

感谢您的支持与鼓励

支付宝扫一扫打赏

微信扫一扫打赏