一、前置知识点
二、部署Etcd群集
三、安装Docker
四、部署Master Node
五、部署Worker Node
六、部署Dashboard和CoreDNS
- 6.1 部署Dashboard
- 6.2 部署CoreDNS

一、前置知识点

1.1 生产环境可部署Kubernetes集群的两种方式

目前生产部署kubernetes集群主要有两种方式：

kubeadm

kubeadm是一个kus部署工具，提供kubeadm init和kubeadm join，用于快速部署kubernets集群。

官方地址

二进制包

从github下载发行版的二进制包，手动部署每个组件，组成kubernetes集群。

kubeadm降低部署门槛，但屏蔽了很多细节，遇到问题很难排查原因。如果想要更容易可控，推荐使用二进制包部署k8s集群，虽然手动部署麻烦点，但是期间可以学习hang多工作原理，也利于后期维护。

1.2 安装要求

开始之前，部署k8s集群机器需要满足一下几个条件：

一台或多台机器，操作系统CentOS7.x-86_x64
硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多
可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载好镜像包上传到服务器节点
禁止swap分区

1.3 准备环境

所有二进制包：

链接:https://pan.baidu.com/s/1WxoA_GjaNgJqR1hmXPyzcg 提取码:bryx

软件环境：

软件	版本
操作系统	CentOS7.8_x64(mini)
Docker	19-ce
kubernetes	1.18.3

服务器整体规划：

角色	IP	组件
k8s-master01	192.168.171.131	Kube-apiserver,kube-controller-manager, kube-scheduler, etcd
K8s-master02	192.168.171.134	Kube-apiserver,kube-controller-manager, kube-scheduler
K8s-node01	192.168.171.132	kubelet，kube-proxy，docker etcd
k8s-node02	192.168.171.133	kubelet，kube-proxy，docker etcd
Load Balancer（Master）	192.168.171.141，192.168.171.88(vip)	Nginx L4
Load Balancer（Backup）	192.168.171.142	Nginx L4

注意：考虑到有些朋友电脑配置较低，这么多虚拟机带不动，所以这一套高可用集群分两部实施，先部署一套单Master架构（192.168.171.131/132/133），再扩容多Master架构，顺便熟悉一下Master扩容流程。

单Master架构图：

单Master服务器规划：

角色	IP	组件
k8s-master01	192.168.171.131	kube-apiserver，kube-controller-manager，kube-scheduler，etcd
k8s-node01	192.168.171.132	kubelet，kube-proxy，docker etcd
k8s-node02	192.168.171.133	kubelet，kube-proxy，docker etcd

1.4 初始化操作系统配置

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
setenforce 0  # 临时

# 关闭swap
swapoff -a  # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

# 根据规划设置主机名
hostnamectl set-hostname <hostname>

# 在master添加hosts
cat >> /etc/hosts << EOF
192.168.171.131 k8s-master01
192.168.171.132 k8s-node01
192.168.171.133 k8s-node02
EOF

# 开启ipvs支持
iptables -P FORWARD ACCEPT
yum -y install ipvsadm  ipset
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF


# 将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
EOF
sysctl --system  # 生效

# 时间同步
yum install ntpdate -y
ntpdate time.windows.com

二、部署Etcd群集

Etcd是一个分布式键值存储系统，kubernetes使用Etcd存储，福偶一先准备一个Etcd数据库群集，这里使用3台组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可以容忍两台机器故障。

节点名称	IP
etcd01	192.168.171.131
etcd02	192.168.171.132
etcd03	192.168.171.133

注：为了节省机器，这里与k8s节点机器复用，也可以独立于k8s集群之外部署，只要apiserver能丽娜姐上就行。

2.1 准备cfssl证书生成工具

cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl封方便使用。

找任意一台服务器操作，这里用Master节点。

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.2 生成Etcd证书

2.2.1 自签证书颁发机构（CA）

创建工作目录：

mkdir -p ~/TLS/{etcd,k8s}

cd ~/TLS/etcd

自签CA：

ca-config.json

{
    "signing": {
      "default": {
        "expiry": "87600h"
      },
      "profiles": {
        "www": {
           "expiry": "87600h",
           "usages": [
              "signing",
              "key encipherment",
              "server auth",
              "client auth"
          ]
        }
      }
    }
  }

vim ca-csr.json

{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}

生成证书：

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

ls *pem
# ca-key.pem  ca.pem

2.2.2 使用自签CA签发Etcd HTTPS证书

创建证书申请文件server-csr.json:

{
    "CN": "etcd",
    "hosts": [
    "192.168.171.131",
    "192.168.171.132",
    "192.168.171.133"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

注:上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，一个都不能少，为了方便后期扩容可以多写几个预留IP。或者可以直接空这hosts：[]

生成证书:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

ls server*pem
# server-key.pem  server.pem

2.3 从github下载二进制文件

wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

2.4 部署Etcd集群

一下在节点1上操作，为简化操作，待会节点1生成的所有文件拷贝到节点2和节点3。

2.4.1 创建工作目录并解压二进制包

mkdir /opt/etcd/{bin,cfg,ssl} -p
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

2.4.2 创建etcd配置文件

vim /opt/etcd/cfg/etcd.conf

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.171.131:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.171.131:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.171.131:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.171.131:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.171.131:2380,etcd02=https://192.168.171.132:2380,etcd03=https://192.168.171.133:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新建集群，existing表示加入已有集群

2.4.3 systemd管理etcd

vim /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem --logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

2.4.4 拷贝刚才生成的证书

把刚才生成的证书拷贝到配置文件中的路径中

cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

2.4.5 启动并设置开机启动

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

2.4.6 将上边节点1所有生成的文件拷贝到节点2和节点3

scp -r /opt/etcd/ root@192.168.171.132:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.171.132:/usr/lib/systemd/system/
scp -r /opt/etcd/ root@192.168.171.133:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.171.133:/usr/lib/systemd/system/

然后再节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP:

vim /opt/etcd/cfg/etcd.conf

#[Member]
ETCD_NAME="etcd01" # 修改此处，节点2改为etcd02，节点3改为etcd03
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.171.131:2380" # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.171.131:2379" # 修改此处为当前服务器IP

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.171.131:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.171.131:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.171.131:2380,etcd02=https://192.168.171.132:2380,etcd03=https://192.168.171.133:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

最后启动etcd并设置开机启动，同上。

2.4.7 查看集群状态

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.171.131:2379,https://192.168.171.132:2379,https://192.168.171.133:2379" endpoint health

# https://192.168.171.131:2379 is healthy: successfully committed proposal: took = 8.154404ms
# https://192.168.171.133:2379 is healthy: successfully committed proposal: took = 9.044117ms
# https://192.168.171.132:2379 is healthy: successfully committed proposal: took = 10.000825ms

ETCDCTL_API=3：表示使用etcd的v3版本的API接口

如果输出上面信息，就说明集群部署成功。如果有问题第一步先看日志：/var/log/message 或 journalctl -u etcd

三、安装Docker

下载Docker

wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

以下在所有节点操作，这里采用二进制安装，用yum安装也是一样的。

3.1 解压二进制报

tar zxvf docker-19*

mv docker/ /usr/bin

3.2 systemd 管理docker

vim /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

3.3 创建配置文件

mkdir /etc/docker
vim /etc/docker/daemon.json

{
  "registry-mirrors": ["https://6o9frx9d.mirror.aliyuncs.com"],
   "exec-opts": ["native.cgroupdriver=systemd"],
   "log-driver": "json-file",
   "log-opts": {
      "max-size": "100m"
    },
    "storage-driver": "overlay2"
}

registry-mirrors 阿里云镜像加速器

四、部署Master Node

4.1 生成kube-apiserver证书

4.1.1 自签证书颁发机构（CA）

vim ~/TLS/k8s/ca-config.json

{
    "signing": {
      "default": {
        "expiry": "87600h"
      },
      "profiles": {
        "kubernetes": {
           "expiry": "87600h",
           "usages": [
              "signing",
              "key encipherment",
              "server auth",
              "client auth"
          ]
        }
      }
    }
  }

vim ~/TLS/k8s/ca-csr.json

{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

生成证书：

cd ~/TLS/k8s
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

ls *pem
# ca-key.pem  ca.pem

4.1.2 使用自签CA签发kub-apiserver HTTPS证书

创建证书申请文件：

vim ~/TLS/k8s/server-csr.json

{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.171.131",
      "192.168.171.132",
      "192.168.171.133",
      "192.168.171.134",
      "192.168.171.135",
      "192.168.171.136",
      "192.168.171.137",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local",
      "k8s-master01",
      "k8s-master02",
      "k8s-master03",
      "k8s-node01",
      "k8s-node02",
      "k8s-node03",
      "etcd01",
      "etcd02",
      "etcd03"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

注：上述文件hosts字段中IP为所有Master/LB/VIP/IP，一个都不能少，为了方便后期维护扩展可以多写几个预留的IP。

生成证书：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

ls server*pem
# server-key.pem  server.pem

4.2 从github下载二进制文件

下载地址

注：打开连接你会发现里面有很多包，下载一个server包就够了，包含了Master和Worker Node二进制文件。

wget https://dl.k8s.io/v1.18.3/kubernetes-server-linux-amd64.tar.gz

4.3 解压二进制包

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 

tar zxvf kubernetes-server-linux-amd64.tar.gz

cd kubernetes/server/bin

cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

4.4 部署kube-apiserver

4.4.1 创建配置文件

vim /opt/kubernetes/cfg/kube-apiserver.conf

KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://192.168.171.131:2379,https://192.168.171.132:2379,https://192.168.171.133:2379 \
--bind-address=192.168.171.131 \
--secure-port=6443 \
--advertise-address=192.168.171.131 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

-–logtostderr：启用日志
--v：日志等级
--log-dir：日志目录
--etcd-servers：etcd集群地址
–-bind-address：监听地址
–-secure-port：https安全端口
–-advertise-address：集群通告地址
–-allow-privileged：启用授权
–-service-cluster-ip-range：Service虚拟IP地址段
–-enable-admission-plugins：准入控制模块
–-authorization-mode：认证授权，启用RBAC授权和节点自管理
–-enable-bootstrap-token-auth：启用TLS bootstrap机制
–-token-auth-file：bootstrap token文件
–-service-node-port-range：Service nodeport类型默认分配端口范围
–-kubelet-client-xxx：apiserver访问kubelet客户端证书
–-tls-xxx-file：apiserver https证书
–-etcd-xxxfile：连接Etcd集群证书
–-audit-log-xxx：审计日志

4.4.2 拷贝刚才生成的证书

把刚才生成的证书拷贝到配置文件中的路径

cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

4.4.3 启用TLS Bootstrapping机制

TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程， Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。

TLS bootstraping 工作流程：

token自行生成：

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

# c47ffb939f5ca36231d9e3121a252940

创建上述配置文件中token文件：

vim /opt/kubernetes/cfg/token.csv

c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"

格式：token，用户名，UID，用户组

4.4.4 systemd管理apiserver

vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

4.4.5 启动并设置开机启动

systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver

4.4.6 授权kubelet-bootstrap用户允许请求证书

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

4.5 部署kube-controller-manager

4.5.1 创建配置文件

vim /opt/kubernetes/cfg/kube-controller-manager.conf

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"

–master：通过本地非安全本地端口8080连接apiserver。
–leader-elect：当该组件启动多个时，自动选举（HA）
–cluster-signing-cert-file/–cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

4.5.2 systemd管理controller-manager

vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

4.5.3 启动并设置开机启动

systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

4.6 部署kube-scheduler

4.6.1 创建配置文件

vim /opt/kubernetes/cfg/kube-scheduler.conf

KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"

–master：通过本地非安全本地端口8080连接apiserver。
–leader-elect：当该组件启动多个时，自动选举（HA）

4.6.2 systemd管理scheduler

vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target

4.6.3 启动并设置开机启动

systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler

4.6.4 查看集群状态

所有组件都已经启动成功，通过kubectl工具查看当前集群组件状态：

kubectl get cs

NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
etcd-2               Healthy   {"health":"true"}

如上输出说明Master节点组件运行正常。

五、部署Worker Node

下面还是在Master Node上操作，即同时作为Worker Node

5.1 创建工作目录并拷贝二进制文件

在所有worker node创建工作目录

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

从master节点拷贝：

cd ~/kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes/bin   # 本地拷贝

5.2 部署kubelet

5.2.1 创建配置文件

vim /opt/kubernetes/cfg/kubelet.conf

KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=k8s-master01 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause-amd64:3.0"

–-hostname-override：显示名称，集群中唯一
–-network-plugin：启用CNI
–-kubeconfig：空路径，会自动生成，后面用于连接apiserver
-–bootstrap-kubeconfig：首次启动向apiserver申请证书
-–config：配置参数文件
-–cert-dir：kubelet证书生成目录
–-pod-infra-container-image：管理Pod网络容器的镜像

5.2.2 配置文件参数

vim /opt/kubernetes/cfg/kubelet-config.yml

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
        - 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
          anonymous:
                  enabled: false
          webhook:
                  cacheTTL: 2m0s
                  enabled: true
          x509:
                  clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
        mode: Webhook
        webhook:
                cacheAuthorizedTTL: 5m0s
                cacheUnauthorizedTTL: 30s
evictionHard:
        imagefs.available: 15%
        memory.available: 100Mi
        nodefs.available: 10%
        nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110

5.2.3 生成bootstrap.kubeconfig文件

执行命令

KUBE_APISERVER="https://192.168.171.131:6443" # apiserver IP:PORT
TOKEN="c47ffb939f5ca36231d9e3121a252940" # 与token.csv里保持一致

cd ~/
# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true  --server=${KUBE_APISERVER}  --kubeconfig=bootstrap.kubeconfig

kubectl config set-credentials "kubelet-bootstrap"  --token=${TOKEN} --kubeconfig=bootstrap.kubeconfig

kubectl config set-context default  --cluster=kubernetes --user="kubelet-bootstrap"  --kubeconfig=bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

拷贝到配置文件路径：

cp bootstrap.kubeconfig /opt/kubernetes/cfg

5.2.4 systemd管理kubelet

vim /usr/lib/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
After=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

5.2.5 启动并设置开机启动

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

5.3 批准kubelet证书申请并加入集群

# 查看kubelet证书请求
kubectl get csr
NAME                                                   AGE    SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A   6m3s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# 批准申请
kubectl certificate approve node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A

# 查看节点
kubectl get node
NAME         STATUS     ROLES    AGE   VERSION
k8s-master01   NotReady   <none>   7s    v1.18.3

注：由于网络插件还没有部署，节点会没有准备就绪 NotReady

5.4 部署kube-proxy

5.4.1 创建配置文件

vim /opt/kubernetes/cfg/kube-proxy.conf

KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--config=/opt/kubernetes/cfg/kube-proxy-config.yml \
--masquerade-all=true"

5.4.2 配置参数文件

vim /opt/kubernetes/cfg/kube-proxy-config.yml

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
        kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master01
clusterCIDR: 10.0.0.0/24
mode: ipvs

5.4.3 生成kube-proxy.kubeconfig文件

生成kube-proxy：

cd ~/TLS/k8s
vim kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "K8S",
      "OU": "Systrm"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy


ls kube-proxy*pem
# kube-proxy-key.pem  kube-proxy.pem

生成kubeconfig文件：

KUBE_APISERVER="https://192.168.171.131:6443"
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

拷贝到配置文件指定路径

cp kube-proxy.kubeconfig /opt/kubernetes/cfg/

5.4.4 systemd管理kube-proxy

vim /usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

5.4.5 启动并设置开机启动

systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy

5.5 部署CNI网络

先准备好CNI二进制文件：

wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz

解压二进制包并移动到默认工作目录：

mkdir /opt/cni/bin
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

部署CNI网络：

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl apply -f kube-flannel.yml

kubectl get pods -n kube-system
NAME                          READY   STATUS    RESTARTS   AGE
kube-flannel-ds-amd64-2pc95   1/1     Running   0          72s

kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    <none>   41m   v1.18.3

部署好网络插件，Node准备就绪。

5.6 授权apiserver访问kubelet

vim apiserver-to-kubelet-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes

kubectl apply -f apiserver-to-kubelet-rbac.yaml

5.7 新增加Worker Node

5.7.1 拷贝已部署好的Node相关文件到新节点

在master节点将node涉及到的文件拷贝到新节点192.168.171.132/133

scp -r /opt/kubernetes root@192.168.171.132:/opt/

scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.171.132:/usr/lib/systemd/system

scp -r /opt/cni/ root@192.168.171.132:/opt/

scp /opt/kubernetes/ssl/ca.pem root@192.168.171.132:/opt/kubernetes/ssl

5.7.2 删除kubelet证书和kubeconfig文件

rm /opt/kubernetes/cfg/kubelet.kubeconfig 
rm -f /opt/kubernetes/ssl/kubelet*

注：这几个文件是证书申请审批后自动生成的，每个Node不同，必须删除重新生成。

5.7.3 修改主机名

vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node01

vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node01

5.7.4 设置开机启动

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy

5.7.5 在Master上批准新Node kubelet证书申请

kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro   89s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

kubectl certificate approve node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro

5.7.6 查看Node状态

kubectl get node
NAME         STATUS     ROLES    AGE   VERSION
k8s-master01   Ready      <none>   65m   v1.18.3
k8s-node01    Ready      <none>   12m   v1.18.3
k8s-node02    Ready      <none>   81s   v1.18.3

Node2（192.168.171.133 ）节点同上。记得修改主机名！

六、部署Dashboard和CoreDNS

6.1 部署Dashboard

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部：

vi recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

kubectl apply -f recommended.yaml
kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS              RESTARTS   AGE
pod/dashboard-metrics-scraper-694557449d-z8gfb   1/1     Running             0          2m18s
pod/kubernetes-dashboard-9774cc786-q2gsx         1/1     Running             0          2m19s

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.141   <none>        8000/TCP        2m19s
service/kubernetes-dashboard        NodePort    10.0.0.239   <none>        443:30001/TCP   2m19s

访问地址：https://NodeIP:30001

创建service account并绑定默认cluster-admin管理员集群角色：

kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

使用输出的token登录Dashboard。

6.2 部署CoreDNS

CoreDNS用于集群内部Service名称解析。

kubectl apply -f coredns.yaml

kubectl get pods -n kube-system 
NAME                          READY   STATUS    RESTARTS   AGE
coredns-5ffbfd976d-j6shb      1/1     Running   0          32s
kube-flannel-ds-amd64-2pc95   1/1     Running   0          38m
kube-flannel-ds-amd64-7qhdx   1/1     Running   0          15m
kube-flannel-ds-amd64-99cr8   1/1     Running   0          26m

至此，单Master集群部署完成，下一篇扩容为多Master集群~

一、前置知识点

1.1 生产环境可部署Kubernetes集群的两种方式

1.2 安装要求

1.3 准备环境

1.4 初始化操作系统配置

二、部署Etcd群集

2.1 准备cfssl证书生成工具

2.2 生成Etcd证书

2.2.1 自签证书颁发机构（CA）

2.2.2 使用自签CA签发Etcd HTTPS证书

2.3 从github下载二进制文件

2.4 部署Etcd集群

2.4.1 创建工作目录并解压二进制包

2.4.2 创建etcd配置文件

2.4.3 systemd管理etcd

2.4.4 拷贝刚才生成的证书

2.4.5 启动并设置开机启动

2.4.6 将上边节点1所有生成的文件拷贝到节点2和节点3

2.4.7 查看集群状态

三、安装Docker

3.1 解压二进制报

3.2 systemd 管理docker

3.3 创建配置文件

四、部署Master Node

4.1 生成kube-apiserver证书

4.1.1 自签证书颁发机构（CA）

4.1.2 使用自签CA签发kub-apiserver HTTPS证书

4.2 从github下载二进制文件

4.3 解压二进制包

4.4 部署kube-apiserver

4.4.1 创建配置文件

4.4.2 拷贝刚才生成的证书

4.4.3 启用TLS Bootstrapping机制

4.4.4 systemd管理apiserver

4.4.5 启动并设置开机启动

4.4.6 授权kubelet-bootstrap用户允许请求证书

4.5 部署kube-controller-manager

4.5.1 创建配置文件

4.5.2 systemd管理controller-manager

4.5.3 启动并设置开机启动

4.6 部署kube-scheduler

4.6.1 创建配置文件

4.6.2 systemd管理scheduler

4.6.3 启动并设置开机启动

4.6.4 查看集群状态

五、部署Worker Node

5.1 创建工作目录并拷贝二进制文件

5.2 部署kubelet

5.2.1 创建配置文件

5.2.2 配置文件参数

5.2.3 生成bootstrap.kubeconfig文件

5.2.4 systemd管理kubelet

5.2.5 启动并设置开机启动

5.3 批准kubelet证书申请并加入集群

5.4 部署kube-proxy

5.4.1 创建配置文件

5.4.2 配置参数文件

5.4.3 生成kube-proxy.kubeconfig文件

5.4.4 systemd管理kube-proxy

5.4.5 启动并设置开机启动

5.5 部署CNI网络

5.6 授权apiserver访问kubelet

5.7 新增加Worker Node

5.7.1 拷贝已部署好的Node相关文件到新节点

5.7.2 删除kubelet证书和kubeconfig文件

5.7.3 修改主机名

5.7.4 设置开机启动

5.7.5 在Master上批准新Node kubelet证书申请

5.7.6 查看Node状态

六、部署Dashboard和CoreDNS

6.1 部署Dashboard

6.2 部署CoreDNS

发表评论

共 0 条评论