02 搭建一个完整的Kubernetes集群

生产环境K8S平台规划

单Master集群

一个master,存在单点故障，master节点出现问题无法创建和管理应用,已经部署的还是可以工作
etcd可以独立部署(集群部署),只要k8s能连接etcd即可

多Master集群（HA）

多个master如何工作,前面使用负载均衡
重点在apiserver ，node改成连接LB负载均衡
所有请求不会直接到某个 apiserver中，而是先到LB再分配

测试环境使用单master集群
- 单master 2个node
- etc集群部署分别部署在3台里

生产环境必须使用多master,生产环境绝对不允许出现单点故障
- 按照下面的生产环境准备,node横向扩容
- 复用etcd的话至少6台
- 机器多 etcd也独立部署
- node 配置要高

生产环境 K8S 平台规划

服务器硬件配置推荐

node 不够可以水平扩展
master很少扩展,2-3台就足够了
集群中每台服务器至少预留30%的资源,跑满80%的资源就要考虑扩展

官方提供的三种部署方式

minikube
- Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，仅用于尝试Kubernetes或日常开发的用户使用。
- 部署地址：https://kubernetes.io/docs/setup/minikube/

kubeadm
- Kubeadm也是一个工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。
- 部署地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

二进制
- 推荐，从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。
- 下载地址：https://github.com/kubernetes/kubernetes/releases

单master集群搭建

系统初始化

关闭防火墙：
systemctl stop firewalld
systemctl disable firewalld

关闭selinux：
setenforce 0 # 临时
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久

关闭swap：
swapoff -a  # 临时
vim /etc/fstab  # 永久 注释掉swap那项，阿里云没有，只有ext4

同步系统时间：
# ntpdate time.windows.com

添加hosts：
# vim /etc/hosts
172.17.70.245 k8s-master2
172.17.70.246 k8s-master
172.17.70.247 k8s-node1
172.17.70.248 k8s-node2

修改主机名：
hostnamectl set-hostname k8s-master

etcd 集群

理解ssl证书

集群中的通信都是基于https进行交互
当前架构中,etcd之间需要https通信,k8s内部master(apiserver) 也需要通信, 需要两条证书
证书分为: 自签证书和权威机构颁发的证书(赛门铁克) 3000元 www.ctnrs.com 泛域名*.ctnrs.com 比较贵
根证书
自签证书不受信任内部服务之间具备一定加密内部之间程序调用不对外，可以使用
证书包括:crt(数字证书) key(私钥) 配置到web服务器
权威机构颁发的证书 https是绿色的安全的，自签是！感叹号不可信任
权威证书如果使用到k8s里，其实也不知道是否可行，因为证书里涉及到可信任的IP列表，可能有影响
https -> CA -> crt|key

生成etcd证书

可在任意节点完成以下操作。

使用cfssl工具自签证书

cfssl 与 openssl一样都可以生成证书
cfssl 使用json文件内容生成，比较直观

1
2
3

[root@k8s-master opt]# ls
TLS.tar.gz
[root@k8s-master opt]# tar -zxvf TLS.tar.gz

[root@k8s-master opt]# cd TLS
[root@k8s-master TLS]# ls -l
total 18820
-rwxr-xr-x 1 root root 10376657 Oct  2 08:37 cfssl
-rwxr-xr-x 1 root root  6595195 Oct  2 08:37 cfssl-certinfo
-rwxr-xr-x 1 root root  2277873 Oct  2 08:37 cfssljson
-rwxr-xr-x 1 root root      344 Oct  3 08:26 cfssl.sh
drwxr-xr-x 2 root root     4096 Oct  3 10:46 etcd   # etcd证书目录
drwxr-xr-x 2 root root     4096 Oct  3 10:46 k8s    # k8s证书目录

# 移动到可执行目录
# cfssl.sh 脚本负责cp,而且他也可以下载,由于我这里已经有脚本了直接cp

[root@k8s-master TLS]# cat cfssl.sh 
#curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
#curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
#curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
cp -rf cfssl cfssl-certinfo cfssljson /usr/local/bin
chmod +x /usr/local/bin/cfssl*

[root@k8s-master TLS]# sh cfssl.sh 
[root@k8s-master TLS]# ls /usr/local/bin/
cfssl  cfssl-certinfo  cfssljson

[root@k8s-master etcd]# cd /opt/TLS/etcd/
[root@k8s-master etcd]# ls -l
-rw-r--r-- 1 root root 287 Oct  3 13:12 ca-config.json              # 相当于CA机构
-rw-r--r-- 1 root root 209 Oct  3 13:12 ca-csr.json                 # CA的请求文件
-rwxr-xr-x 1 root root 178 Oct  3 13:58 generate_etcd_cert.sh       # 执行脚本
-rw-r--r-- 1 root root 306 Oct  3 08:26 server-csr.json             # 配置文件

# 自建CA
# 脚本里有两条命令
[root@k8s-master etcd]# cat generate_etcd_cert.sh 
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

# 单独执行 
# 初始化CA -initca 
# ca的配置文件 ca-csr.json 

[root@k8s-master etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2019/11/06 08:40:34 [INFO] generating a new CA key and certificate from CSR
2019/11/06 08:40:34 [INFO] generate received request
2019/11/06 08:40:34 [INFO] received CSR
2019/11/06 08:40:34 [INFO] generating key: rsa-2048
2019/11/06 08:40:34 [INFO] encoded CSR
2019/11/06 08:40:34 [INFO] signed certificate with serial number 319049982969134642253684014591326256636267413051

# 执行后会生成两个pem文件
[root@k8s-master etcd]# 
[root@k8s-master etcd]# ls -l *.pem
-rw------- 1 root root 1675 Nov  6 08:40 ca-key.pem       # ca的私钥
-rw-r--r-- 1 root root 1265 Nov  6 08:40 ca.pem           # ca的数字证书 可以拿着这两个去办法域名证书

# 为哪个域名去颁发证书
# 通过 server-csr.json
# CN:名字
# hosts是最重要的 可信任的IP 包含所有etcd节点的ip
# key 加密算法和长度
# names 证书属性信息

[root@k8s-master etcd]# vim server-csr.json 

{
    "CN": "etcd",
    "hosts": [
        "172.17.70.246",
        "172.17.70.247",
        "172.17.70.248"
        ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

# 向CA申请证书 
# 指定好ca的证书和私钥，配置文件
# 生成 server开头的证书
# -config=ca-config.json 里面是ca机构的属性 比如域名证书的有效时间 默认的是10年

[root@k8s-master etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

# 会生成server开头的 数字证书和私钥

[root@k8s-master etcd]# ls -l server*.pem
-rw------- 1 root root 1679 Nov  6 08:49 server-key.pem   -> key
-rw-r--r-- 1 root root 1338 Nov  6 08:49 server.pem       -> crt

[root@k8s-master etcd]# ls -l *.pem
-rw------- 1 root root 1675 Nov  6 08:40 ca-key.pem
-rw-r--r-- 1 root root 1265 Nov  6 08:40 ca.pem
-rw------- 1 root root 1679 Nov  6 08:49 server-key.pem
-rw-r--r-- 1 root root 1338 Nov  6 08:49 server.pem

etcd 简介

CoreOS 开源
etcd首先是一个键值存储仓库,用于配置共享和服务发现。
etcd 负责保存 Kubernetes Cluster 的配置信息和各种资源的状态信息。当数据发生变化时，etcd 会快速地通知 Kubernetes 相关组件。
官方推荐奇数节点部署,常见的有3 5 7 分别对应 1 2 3个冗余节点
3台etcd会先选举出1台为主节点,负责写消息,主节点同步给2个从节点。
当主节点挂了,两台从节点会选举出一台新的主节点

1 2	# 介绍文档 https://blog.csdn.net/bbwangj/article/details/82584988

# 优点
1. 简单。使用Go语言编写部署简单；使用HTTP作为接口使用简单；
2. 使用Raft算法保证强一致性让用户易于理解。
3. 数据持久化。etcd默认数据一更新就进行持久化。
4. 安全。etcd支持SSL客户端安全认证。

部署三个 etcd 节点

# 环境
172.17.70.246   k8s-master 	etcd
172.17.70.247   k8s-node1  	etcd
172.17.70.248   k8s-node2  	etcd

# 二进制包下载地址
https://github.com/etcd-io/etcd/releases

# 上传etcd tar包
[root@k8s-master opt]# ls -l etcd.tar.gz 
-rw-r--r-- 1 root root 10148977 Nov  6 09:38 etcd.tar.gz

# 解压
[root@k8s-master opt]# tar -zxvf etcd.tar.gz 

[root@k8s-master opt]# ls
etcd           # 服务目录 放在opt下  
etcd.service   # CentOS7服务启动文件 
# 启动文件里面设置了etct的工作目录 /opt/etcd/bin/etcd 和 配置文件 /opt/etcd/cfg/etcd.conf

# 查看工作目录
[root@k8s-master etcd]# cd /opt/etcd

# 更换etcd版本:可从etcd的github下载最新版本 替换bin下的二进制文件
[root@k8s-master etcd]# ls -l
drwxr-xr-x 2 root root 4096 Oct  2 22:13 bin    # etcd 可执行文件 etcdctl 客户端  
drwxr-xr-x 2 root root 4096 Oct  3 08:32 cfg    # 配置文件
drwxr-xr-x 2 root root 4096 Oct  3 08:36 ssl    # 证书文件 需要将生产的证书加入里面

# 先删除掉 一会替换刚才生成的
[root@k8s-master ssl]# cd /opt/etcd/ssl/
[root@k8s-master ssl]# rm -rf *

# 查看配置文件
# 节点之间通信 2380 需要https通信 拥有证书
# 面向客户端 2379 
# 证书的配置也都在启动服务里 etcd.service 都使用一套证书
# 如果要重装etcd 需要清空下面的目录 ETCD_DATA_DIR="/var/lib/etcd/default.etcd"  

[root@k8s-master etcd]# cd cfg/
[root@k8s-master cfg]# vim etcd.conf 

#[Member]
ETCD_NAME="etcd-1"                                                          # 集群名称 唯一
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"                                  # etcd存储数据目录
ETCD_LISTEN_PEER_URLS="https://172.17.70.246:2380"                          # etcd内部通信 IP+端口
ETCD_LISTEN_CLIENT_URLS="https://172.17.70.246:2379"                        # 客户端监听 IP+端口 程序连接

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.17.70.246:2380"               # 集群内部通信 通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://172.17.70.246:2379"                     # 集群客户端通信
# 集群节点连接配置 名称=ip+端口
ETCD_INITIAL_CLUSTER="etcd-1=https://172.17.70.246:2380,etcd-2=https://172.17.70.247:2380,etcd-3=https://172.17.70.248:2380"
# TOKEN 用于简单认证
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
# 集群状态 新建是new 已有的加入使用 exsiting
ETCD_INITIAL_CLUSTER_STATE="new"

1 2	# 将证书加入到 ssl 里 [root@k8s-master ssl]# cp /opt/TLS/etcd/{ca,server,server-key}.pem ./

# 部署所有节点
[root@k8s-master opt]# scp -r etcd  root@172.17.70.247:/opt
[root@k8s-master opt]# scp -r etcd  root@172.17.70.248:/opt

# 配置文件进入启动服务目录
scp etcd.service root@172.17.70.246:/usr/lib/systemd/system
scp etcd.service root@172.17.70.247:/usr/lib/systemd/system
scp etcd.service root@172.17.70.248:/usr/lib/systemd/system

# 然后在进去修改相应的配置 主要是 ip 和 ETCD_NAME
[root@k8s-node2 etcd]# vim /opt/etcd/cfg/etcd.conf 

#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.17.70.248:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.17.70.248:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.17.70.248:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.17.70.248:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://172.17.70.246:2380,etcd-2=https://172.17.70.247:2380,etcd-3=https://172.17.70.248:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

[root@k8s-node2 etcd]# cat /usr/lib/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
        --name=${ETCD_NAME} \
        --data-dir=${ETCD_DATA_DIR} \
        --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
        --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
        --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
        --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
        --initial-cluster=${ETCD_INITIAL_CLUSTER} \
        --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
        --initial-cluster-state=new \
        --cert-file=/opt/etcd/ssl/server.pem \
        --key-file=/opt/etcd/ssl/server-key.pem \
        --peer-cert-file=/opt/etcd/ssl/server.pem \
        --peer-key-file=/opt/etcd/ssl/server-key.pem \
        --trusted-ca-file=/opt/etcd/ssl/ca.pem \
        --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动etcd

# 三台节点都需要操作
# 第一个几点启动会等待 ， 启动后面的成功再来查看
# 设置开机启动

[root@k8s-master opt]# systemctl daemon-reload
[root@k8s-master opt]# systemctl start etcd
[root@k8s-master opt]# systemctl enable etcd

1 2	# etcd 日志 [root@k8s-master opt]# tailf /var/log/messages

# 查看集群状态 需要指定3个证书哦 
# k8s连接etcd 也需要指定这三个etcd证书

/opt/etcd/bin/etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://172.17.70.246:2379,https://172.17.70.247:2379,https://172.17.70.248:2379" \
cluster-health

# 端口检查
[root@k8s-master opt]# netstat -tnlp|grep etcd
tcp        0      0 172.17.70.246:2379      0.0.0.0:*               LISTEN      1762/etcd           
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      1762/etcd           
tcp        0      0 172.17.70.246:2380      0.0.0.0:*               LISTEN      1762/etcd

部署 Master Node

生成apiserver证书

# 查看目录
[root@k8s-master k8s]# cd /opt/TLS/k8s/
[root@k8s-master k8s]# ls -l
total 20
-rw-r--r-- 1 root root 294 Oct  3 13:12 ca-config.json
-rw-r--r-- 1 root root 263 Oct  3 13:12 ca-csr.json
-rwxr-xr-x 1 root root 321 Oct  3 08:46 generate_k8s_cert.sh
-rw-r--r-- 1 root root 230 Oct  3 13:12 kube-proxy-csr.json          # node上的proxy组件证书
-rw-r--r-- 1 root root 718 Oct  3 08:45 server-csr.json              # apiserver 证书

[root@k8s-master k8s]# vim server-csr.json 
# hosts 最关键了 是否可信任 接口没有浏览器所以需要配置可信任
# 应用程序(服务器IP) 访问 https API (自签证书)
# 1. 证书添加IP可信任 
# 2. 携带CA证书去访问(http转发怎么办)
# 有谁会去访问 LB主从 vip master 写多点没有关系 根据规划 nodeip其实可以不用写

{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",                                  # service第一个IP地址,默认作为apiserver的负载均衡
      "127.0.0.1",                                 # 本地
      "kubernetes",                                # 官方要求 内部为api指定dns名称
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local",
      "172.17.70.245",
      "172.17.70.246",
      "172.17.70.247",
      "172.17.70.248",
      "172.17.70.249",
      "172.17.70.250",
      "172.17.70.251",
      "172.17.70.252",
      "172.17.70.253"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

# 执行脚本 生成证书
[root@k8s-master k8s]# ./generate_k8s_cert.sh 

[root@k8s-master k8s]# ls -l *.pem
-rw------- 1 root root 1675 Nov  6 11:09 ca-key.pem
-rw-r--r-- 1 root root 1359 Nov  6 11:09 ca.pem
-rw------- 1 root root 1679 Nov  6 11:09 kube-proxy-key.pem
-rw-r--r-- 1 root root 1403 Nov  6 11:09 kube-proxy.pem
-rw------- 1 root root 1675 Nov  6 11:09 server-key.pem
-rw-r--r-- 1 root root 1675 Nov  6 11:09 server.pem

部署 apiserve controller-manager和scheduler

在Master节点完成以下操作
配置文件 -> systemd管理组件 -> 启动
二进制包下载地址

1
2
3

# 下载Server Binaries node的也在里面
# 页面打不开 用迅雷下载
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#downloads-for-v1161

# 将需要使用的包 替换 如master节点
[root@k8s-master bin]# cd /opt/kubernetes/bin/
[root@k8s-master bin]# ls
kube-apiserver  kube-controller-manager  kubectl  kube-scheduler

解压 k8s-master.tar.gz

1 2	[root@k8s-master opt]# ls -l /opt/k8s-master.tar.gz [root@k8s-master bin]# tar -zxvf k8s-master.tar.gz

# 服务启动文件 注意服务配置的目录
# 我这里是放在 opt下

[root@k8s-master opt]# cat kube-apiserver.service 
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

[root@k8s-master opt]# tree kubernetes/
kubernetes/
├── bin
│   ├── kube-apiserver
│   ├── kube-controller-manager
│   ├── kubectl
│   └── kube-scheduler
├── cfg
│   ├── kube-apiserver.conf
│   ├── kube-controller-manager.conf
│   ├── kube-scheduler.conf
│   └── token.csv
├── logs  # 日志目录
└── ssl   # 证书目录

配置证书

# copy证书
[root@k8s-master ssl]# cd /opt/kubernetes/ssl/
[root@k8s-master ssl]# cp /opt/TLS/k8s/*.pem .
[root@k8s-master ssl]# ls
ca-key.pem  ca.pem  kube-proxy-key.pem  kube-proxy.pem  server-key.pem  server.pem
# 保留server证书即可 kube-proxy 是给node用的
[root@k8s-master ssl]# rm -rf kube-proxy*
[root@k8s-master ssl]# ls
ca-key.pem  ca.pem  server-key.pem  server.pem

配置文件

1 2	# 官方文档 https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/

kube-apiserver.conf

[root@k8s-master cfg]# vim kube-apiserver.conf 

KUBE_APISERVER_OPTS="--logtostderr=false \
# 日志级别 从小到大 一般情况下2 越大信息越多
--v=2 \
# 日志目录
--log-dir=/opt/kubernetes/logs \
# 连接etcd
--etcd-servers=https://192.168.31.61:2379,https://192.168.31.62:2379,https://192.168.31.63:2379 \
# 监听IP地址
--bind-address=192.168.31.61 \
# 监听端口
--secure-port=6443 \
# 通告地址 让node连接 内网
--advertise-address=192.168.31.61 \
# 允许创建的容器权限 超级权限创建容器
--allow-privileged=true \
# service 为pod 做虚拟ip访问网段
--service-cluster-ip-range=10.0.0.0/24 \
# 插件
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
# 授权模式 
--authorization-mode=RBAC,Node \
# 自动请求颁发证书
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
# 暴露端口范围
--service-node-port-range=30000-32767 \
# 连接 kubelet 证书
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
# 配置apiserver使用https证书
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
# etcd 证书 
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
# 日志审计 访问记录 
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

kube-controller-manager.conf

1 2	# 官方文档 https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

[root@k8s-master cfg]# vim kube-controller-manager.conf 

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
# 连接master ip 8080是apiserver监听端口 默认的
--master=127.0.0.1:8080 \
# 监听本地 不对外
--address=127.0.0.1 \
--allocate-node-cidrs=true \
# POD IP段
--cluster-cidr=10.244.0.0/16 \
# service IP段
--service-cluster-ip-range=10.0.0.0/24 \
# 给 node 证书 
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
# 给node证书 的有效期时间
--experimental-cluster-signing-duration=87600h0m0s"

kube-scheduler.conf

[root@k8s-master cfg]# vim kube-scheduler.conf 

KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
# 参加选举
--leader-elect \
--master=127.0.0.1:8080 \
--address=127.0.0.1"

``` 

#### 修改好配置文件

1. 只需要修改 apiserver.conf 
2. 剩下的两个服务都是连接本地

```html
[root@k8s-master cfg]# vim kube-apiserver.conf 

KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://172.17.70.246:2379,https://172.17.70.247:2379,https://172.17.70.248:2379 \
--bind-address=172.17.70.246 \
--secure-port=6443 \
--advertise-address=172.17.70.246 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

启动服务

1
2
3

# 启动服务加载 
[root@k8s-master opt]# cd /opt/
[root@k8s-master opt]# mv kube-apiserver.service kube-controller-manager.service kube-scheduler.service /usr/lib/systemd/system

systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
systemctl enable kube-apiserver
systemctl enable kube-controller-manager
systemctl enable kube-scheduler

[root@k8s-master opt]# systemctl start kube-apiserver
[root@k8s-master opt]# ps -ef|grep kube

# 查看日志
[root@k8s-master opt]# cd /opt/kubernetes/logs/
[root@k8s-master logs]# ls -l
total 112
lrwxrwxrwx 1 root root    61 Nov  6 12:07 kube-apiserver.ERROR -> kube-apiserver.k8s-master.root.log.ERROR.20191106-120708.2088
lrwxrwxrwx 1 root root    60 Nov  6 12:07 kube-apiserver.INFO -> kube-apiserver.k8s-master.root.log.INFO.20191106-120705.2088
-rw-r--r-- 1 root root   414 Nov  6 12:07 kube-apiserver.k8s-master.root.log.ERROR.20191106-120708.2088
-rw-r--r-- 1 root root 90528 Nov  6 12:07 kube-apiserver.k8s-master.root.log.INFO.20191106-120705.2088
-rw-r--r-- 1 root root  1356 Nov  6 12:07 kube-apiserver.k8s-master.root.log.WARNING.20191106-120706.2088
lrwxrwxrwx 1 root root    63 Nov  6 12:07 kube-apiserver.WARNING -> kube-apiserver.k8s-master.root.log.WARNING.20191106-120706.2088

[root@k8s-master logs]# less /opt/kubernetes/logs/kube-apiserver.INFO

1 2	# 将kubectl 放入/usr/bin 方便使用 [root@k8s-master cfg]# mv /opt/kubernetes/bin/kubectl /usr/local/bin

# 查看集群状态 过会回更新成功 能看到即可 
[root@k8s-master local]# kubectl get cs
NAME                 AGE
scheduler            <unknown>
controller-manager   <unknown>
etcd-0               <unknown>
etcd-2               <unknown>
etcd-1               <unknown>

启用 TLS Bootstrapping

[root@k8s-master cfg]# cat kube-apiserver.conf |grep bootstrap
--enable-bootstrap-token-auth=true \

[root@k8s-master cfg]# cat /opt/kubernetes/cfg/token.csv 
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
# 格式：token,用户,uid,用户组
# 当机器多的时候 会自动给 kubelet派发证书  node带着这个token 即可 

# 给kubelet-bootstrap授权：
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

# 提示成功 clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

# token也可自行生成替换：
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
但apiserver配置的token必须要与node节点bootstrap.kubeconfig配置里一致。

部署 Worker Node

部署docker

1	Docker 二进制包下载地址：https://download.docker.com/linux/static/stable/x86_64/

[root@k8s-master opt]# scp k8s-node.tar.gz root@172.17.70.247:/opt
[root@k8s-node1 opt]# tar -zxvf k8s-node.tar.gz 
[root@k8s-node1 opt]# ls -l
total 207880
-rw-r--r-- 1 root root  36662740 Aug 15 19:33 cni-plugins-linux-amd64-v0.8.2.tgz   # 插件
-rw-r--r-- 1 root root       110 Oct  3 10:01 daemon.json                          # docker 配置文件
-rw-r--r-- 1 root root  48047231 Jun 25 16:45 docker-18.09.6.tgz                   # docker 二进制
-rw-r--r-- 1 root root       501 Oct  3 10:01 docker.service                       # docker 启动
drwxr-xr-x 5 root root      4096 Nov  6 10:04 etcd
-rw-r--r-- 1 root root 128129460 Nov  6 15:32 k8s-node.tar.gz
-rw-r--r-- 1 root root       268 Oct  2 23:11 kubelet.service                      # 启动服务
-rw-r--r-- 1 root root       253 Oct  2 23:11 kube-proxy.service                   # 启动服务
drwxr-xr-x 6 root root      4096 Oct  2 22:14 kubernetes                           # node目录

# 二进制安装docker
[root@k8s-node1 opt]# tar -zxvf docker-18.09.6.tgz 
docker/
docker/docker
docker/docker-init
docker/ctr
docker/docker-proxy
docker/runc
docker/containerd
docker/dockerd
docker/containerd-shim

[root@k8s-node1 opt]# mv docker/* /usr/bin/
[root@k8s-node1 opt]# mv docker.service /usr/lib/systemd/system

# docker服务启动文件
[root@k8s-node1 opt]# cat /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

# 配置文件
[root@k8s-node1 opt]# mkdir -p /etc/docker
[root@k8s-node1 opt]# mv daemon.json /etc/docker/

[root@k8s-node1 opt]# systemctl start docker
[root@k8s-node1 opt]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

[root@k8s-node1 opt]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.14.4.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.695GiB
Name: k8s-node1
ID: QXH6:OCKS:MDUA:JW6M:IBUW:IWM4:JJ3L:6MLR:DCNO:755D:GEGP:NEIF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 192.168.31.70
 127.0.0.0/8
Registry Mirrors:
 http://bc437cce.m.daocloud.io/
Live Restore Enabled: false
Product License: Community Engine

配置 kubelet 和 kube-proxy

[root@k8s-node1 opt]# tree kubernetes/
kubernetes/
├── bin
│   ├── kubelet
│   └── kube-proxy
├── cfg
│   ├── bootstrap.kubeconfig
│   ├── kubelet.conf
│   ├── kubelet-config.yml
│   ├── kube-proxy.conf
│   ├── kube-proxy-config.yml
│   └── kube-proxy.kubeconfig
├── logs
└── ssl

# conf 基本配置文件
# kubeconfig 连接apiserver的配置文件
# yml 主要配置文件 早起版本没有 都是conf里 动态更新配置文件beta

[root@k8s-node1 cfg]# cd /opt/kubernetes/cfg

[root@k8s-node1 cfg]# vim kubelet.conf 

KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
# 当前注册到k8s里的名称 唯一
--hostname-override=k8s-node1 \
# 网络插件
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
# 目录证书路径
--cert-dir=/opt/kubernetes/ssl \
# 这个是一个基础容器，每一个Pod启动的时候都会启动一个这样的容器。
# 现在每个版本的Kubernetes都把这个镜像打包，你可以提前传到自己的registry上，然后再用这个参数指定。
# registry.aliyuncs.com/archon/pause-amd64:3.0
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"

# bootstrap 是 k8s为了解决kubelet 很多 手动办法证书,自动能为加入到集群的node 办法证书 
# 用于连接apiserver

[root@k8s-node1 cfg]# vim bootstrap.kubeconfig 

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    # master地址
    server: https://172.17.70.246:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    # 必须和master上的token一致 cat /opt/kubernetes/cfg/token.csv 
    # 是用这个token来请求 颁发证书
    token: c47ffb939f5ca36231d9e3121a252940

[root@k8s-node1 cfg]# vim kube-proxy.conf 

KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"

# 连接apiserver
# 每个组件都会连接apiserver

[root@k8s-node1 cfg]# vim kube-proxy.kubeconfig 

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    # apiserver地址 master地址
    server: https://172.17.70.246:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kube-proxy
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
  user:
    client-certificate: /opt/kubernetes/ssl/kube-proxy.pem
    client-key: /opt/kubernetes/ssl/kube-proxy-key.pem

[root@k8s-node1 cfg]# vim kube-proxy-config.yml 
# 动态调整proxy配置

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
address: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-node1
clusterCIDR: 10.0.0.0/24
mode: ipvs
ipvs:
  scheduler: "rr"
iptables:
  masqueradeAll: true

配置启动文件

1	[root@k8s-node1 opt]# mv *.service /usr/lib/systemd/system

证书拷贝

1
2
3

[root@k8s-master k8s]# cd /opt/TLS/k8s/
[root@k8s-master k8s]# scp ca.pem ca-key.pem kube-proxy*.pem root@172.17.70.247:/opt/kubernetes/ssl/
[root@k8s-master k8s]# scp ca.pem ca-key.pem kube-proxy*.pem root@172.17.70.248:/opt/kubernetes/ssl/

启动服务

[root@k8s-node1 cfg]# systemctl start kubelet 
[root@k8s-node1 cfg]# systemctl enable kubelet 

systemctl start kubelet
systemctl start kube-proxy
systemctl enable kubelet
systemctl enable kube-proxy

# 需要允许给node办法证书

[root@k8s-node1 cfg]# tail /opt/kubernetes/logs/kubelet.INFO 
I1106 16:28:34.120488    2153 feature_gate.go:216] feature gates: &{map[]}
I1106 16:28:34.120530    2153 feature_gate.go:216] feature gates: &{map[]}
I1106 16:28:34.549201    2153 mount_linux.go:168] Detected OS with systemd
I1106 16:28:34.549324    2153 server.go:410] Version: v1.16.0
I1106 16:28:34.549374    2153 feature_gate.go:216] feature gates: &{map[]}
I1106 16:28:34.549428    2153 feature_gate.go:216] feature gates: &{map[]}
I1106 16:28:34.549508    2153 plugins.go:100] No cloud provider specified.
I1106 16:28:34.549519    2153 server.go:526] No cloud provider specified: "" from the config file: ""

I1106 16:28:34.549545    2153 bootstrap.go:119] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file

I1106 16:28:34.550829    2153 bootstrap.go:150] No valid private key and/or certificate found, reusing existing private key or creating a new one

允许给Node颁发证书

# 查看请求
[root@k8s-master TLS]# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-M69MAprvmbQdWfAwfsB3Zt07_TS-FlDPPiM9WNLCc4k   4m12s   kubelet-bootstrap   Pending

# 允许
[root@k8s-master TLS]# kubectl certificate approve node-csr-M69MAprvmbQdWfAwfsB3Zt07_TS-FlDPPiM9WNLCc4k
certificatesigningrequest.certificates.k8s.io/node-csr-M69MAprvmbQdWfAwfsB3Zt07_TS-FlDPPiM9WNLCc4k approved

# 查看node节点
# 没准备状态 node日志显示还没有部署CNI插件
[root@k8s-master TLS]# kubectl get node
NAME        STATUS     ROLES    AGE   VERSION
k8s-node1   NotReady   <none>   15s   v1.16.0

# 看下node里面的kubelet证书
[root@k8s-node1 ssl]# ls -l kubelet-client*
-rw------- 1 root root 1269 Nov  6 16:34 kubelet-client-2019-11-06-16-34-34.pem
lrwxrwxrwx 1 root root   58 Nov  6 16:34 kubelet-client-current.pem -> /opt/kubernetes/ssl/kubelet-client-2019-11-06-16-34-34.pem

# 帮助我们生成kubelet.kubeconfig
# 它自动生成 帮助我们连接api
# 它使用的证书 就是生成的证书 

[root@k8s-node1 cfg]# cat kubelet.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://172.17.70.246:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
  user:
    client-certificate: /opt/kubernetes/ssl/kubelet-client-current.pem
    client-key: /opt/kubernetes/ssl/kubelet-client-current.pem

# 别忘记还要部署另外一套节点 node2
# 流程一致
# 主要修改的地方: 

# IP地址 bootstrap.kubeconfig kube-proxy.kubeconfig
[root@k8s-node2 cfg]# grep 246 *
bootstrap.kubeconfig:    server: https://172.17.70.246:6443
kube-proxy.kubeconfig:    server: https://172.17.70.246:6443


# 文件中主机名 kubelet.conf kube-proxy-config.yml
[root@k8s-node2 cfg]# grep hostname *
kubelet.conf:--hostname-override=k8s-node2 \
kube-proxy-config.yml:hostnameOverride: k8s-node2

[root@k8s-master k8s]# kubectl get csr
NAME                                                   AGE   REQUESTOR           CONDITION
node-csr-M69MAprvmbQdWfAwfsB3Zt07_TS-FlDPPiM9WNLCc4k   28m   kubelet-bootstrap   Approved,Issued # 已通过
node-csr-_9w0NFGK2oLMrMRWyZbs0UymYNcEtZsNd6imJJmXak4   19s   kubelet-bootstrap   Pending # 请求

[root@k8s-master k8s]# kubectl get node
NAME        STATUS     ROLES    AGE   VERSION
k8s-node1   NotReady   <none>   26m   v1.16.0
k8s-node2   NotReady   <none>   3s    v1.16.0

TLS Bootstrapping 机制流程（kubelet）

部署 CNI 网络 (Flannel)

1	# 二进制包下载地址：https://github.com/containernetworking/plugins/releases

[root@k8s-node1 kubernetes]# tail /opt/kubernetes/logs/kubelet.INFO 
E1106 17:05:08.212430    2153 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
W1106 17:05:09.879829    2153 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

网络没有发现 网络信息在 /etc/cni/net.d下

[root@k8s-node1 opt]# mkdir -p /opt/cni/bin      # 二进制文件目录 kubelet使用 为pod创建网络
[root@k8s-node1 opt]# mkdir -p /etc/cni/net.d    # 生成配置信息目录 
[root@k8s-node1 opt]# tar -zxvf cni-plugins-linux-amd64-v0.8.2.tgz -C /opt/cni/bin

# copy给node2 别忘记创建目录
[root@k8s-node1 opt]# scp -r /opt/cni root@172.17.70.248:/opt
[root@k8s-node2 bin]# mkdir -p /etc/cni/net.d 

# 确保启用cni 
[root@k8s-node1 opt]# cat /opt/kubernetes/cfg/kubelet.conf | grep cni
--network-plugin=cni \ 

# cni只是第三方网络接口 看看这些网络都由哪些
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

部署 flannel

1
2
3

# 在master部署
# 下载 
wget https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

# 上传
[root@k8s-master opt]# vim kube-flannel.yaml 
"Network": "10.244.0.0/16", 
必须要和 cat /opt/kubernetes/cfg/kube-controller-manager.conf 的 一致
--cluster-cidr=10.244.0.0/16 \

# 使用宿主机上的网络
# 关键点 他需要是从网上找镜像 image: lizhenliang/flannel:v0.11.0-amd64
# 我们可以去网上下载好这个镜像 部署到自己仓库中

# 启动 pod 查看pod是否启动
# 会从 lizhenliang/flannel:v0.11.0-amd64 下载镜像有点慢
# 离线部署 提前下载好 导到每个node上本地要是由就不会拉取了


[root@k8s-master opt]# kubectl apply -f kube-flannel.yaml 
[root@k8s-master opt]# kubectl get pods -n kube-system
NAME                          READY   STATUS     RESTARTS   AGE
kube-flannel-ds-amd64-nnrkz   0/1     Init:0/1   0          35s
kube-flannel-ds-amd64-qf5bh   0/1     Init:0/1   0          35s


# 查看pod事件状态 如果是aliyun主机 别忘记node也要出公网啊
[root@k8s-master opt]# kubectl describe pod kube-flannel-ds-amd64-nnrkz -n kube-system

[root@k8s-master opt]# kubectl get pods -n kube-system
NAME                          READY   STATUS    RESTARTS   AGE
kube-flannel-ds-amd64-nnrkz   1/1     Running   0          5m
kube-flannel-ds-amd64-qf5bh   1/1     Running   0          5m

# 没起来试试 
[root@k8s-node2 bin]# systemctl restart kube-proxy

# 查看 node工作状态
[root@k8s-master opt]# kubectl get nodes
NAME        STATUS   ROLES    AGE   VERSION
k8s-node1   Ready    <none>   79m   v1.16.0
k8s-node2   Ready    <none>   52m   v1.16.0

# 查看pod日志 
[root@k8s-master opt]# kubectl logs kube-flannel-ds-amd64-nnrkz -n kube-system
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log kube-flannel-ds-amd64-nnrkz)
# 没有权限 需要授权 通过logs命令可以查看pod日志
[root@k8s-master opt]# vim apiserver-to-kubelet-rbac.yaml 
# 集群角色授权
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log

# 创建
[root@k8s-master opt]# kubectl apply -f apiserver-to-kubelet-rbac.yaml 
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

# 查看pod 日志
[root@k8s-master opt]# kubectl logs kube-flannel-ds-amd64-nnrkz -n kube-system

# 每个node上都会启动个pod
[root@k8s-master opt]# kubectl get pods -n kube-system -o wide
NAME                          READY   STATUS             RESTARTS   AGE   IP              NODE        NOMINATED NODE   READINESS GATES
kube-flannel-ds-amd64-nnrkz   1/1     Running            0          14m   172.17.70.247   k8s-node1   <none>           <none>
kube-flannel-ds-amd64-qf5bh   0/1     CrashLoopBackOff   6          14m   172.17.70.248   k8s-node2   <none>           <none>

# node上会增加flannel 虚拟网卡
# pod通过flannel 跨主机访问 传输数据

创建个nginx pod 试试

[root@k8s-master opt]# kubectl create deployment web --image=nginx:1.16
deployment.apps/web created

[root@k8s-master opt]# kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-866f97c649-hft4w   1/1     Running   0          14s

# 看他分配到哪个节点
[root@k8s-master opt]# kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
web-866f97c649-hft4w   1/1     Running   0          45s   10.244.0.2   k8s-node1   <none>           <none>

# 当创建pod后 会出现cni网桥 虚拟交换机 pod都会加入到这里

# 暴露 到集群外部
[root@k8s-master opt]# kubectl expose deployment web --port=80 --type=NodePort
service/web exposed

[root@k8s-master opt]# kubectl get pods,svc
NAME                       READY   STATUS    RESTARTS   AGE
pod/web-866f97c649-hft4w   1/1     Running   0          5m4s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP        6h6m
service/web          NodePort    10.0.0.161   <none>        80:31618/TCP   46s

# 由于我使用的aliyun  所有我得开端口 
# 为什么两个nodeIP都可以访问呢
http://39.106.100.108:31618/
http://123.56.14.192:31618/

[root@k8s-master opt]# curl 172.17.70.247:31618
[root@k8s-master opt]# curl 172.17.70.248:31618

# 都能访问说明集群网络都正常 单master正常

部署K8S内部DNS服务（CoreDNS）

作用: 对service 提供dns解析

[root@k8s-master ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP        60m
web          NodePort    10.0.0.213   <none>        80:32023/TCP   16m

1. 部署dns后就可以通过名称访问 转发到pod
2. 没有dns服务 就只能通过内部IP 程序里可能要写死 对变动有影响
3. coredns 是默认dns kubeadm默认安装， 二进制需要单独安装

# 固定的IP 
# 他也是个service
[root@k8s-master ~]# cat coredns.yaml |grep clusterIP
  clusterIP: 10.0.0.2 
  
# 他与node节点上的 地址一致
[root@k8s-node1 cfg]# cat kubelet-config.yml | grep clusterDNS:
# 如果写错了 pod的dns解析有问题

# 部署
[root@k8s-master ~]# kubectl apply -f coredns.yaml 
# 查看pods 命名空间在 kube-system
[root@k8s-master ~]# kubectl get pods -n kube-system
NAME                          READY   STATUS    RESTARTS   AGE
coredns-6d8cfdd59d-vfzxs      1/1     Running   0          48s
kube-flannel-ds-amd64-4jkk8   1/1     Running   7          45m
kube-flannel-ds-amd64-vxfdb   1/1     Running   0          45m

# 创建busybox容器
[root@k8s-master ~]# vim bs.yaml 

apiVersion: v1
kind: Pod
metadata:
    name: busybox
    namespace: default
spec:
    containers:
      - image: busybox:1.28.4
        command:
          - sleep
          - "3600"
        imagePullPolicy: IfNotPresent
        name: busybox
    restartPolicy: Always
    
# 部署
[root@k8s-master ~]# kubectl apply -f bs.yaml 
pod/busybox created

# 查看pods
[root@k8s-master ~]# kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
busybox                1/1     Running   0          11s
web-866f97c649-mksh6   1/1     Running   0          25m

# 看下service
[root@k8s-master ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP        71m
web          NodePort    10.0.0.213   <none>        80:32023/TCP   27m 


# 进入容器 通过servic name 可以ping通,以后就可以直接解析name了 
[root@k8s-master ~]# kubectl exec -it busybox sh
/ # ping 10.0.0.213
PING 10.0.0.213 (10.0.0.213): 56 data bytes
64 bytes from 10.0.0.213: seq=0 ttl=64 time=0.075 ms
64 bytes from 10.0.0.213: seq=1 ttl=64 time=0.038 ms

--- 10.0.0.213 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.038/0.056/0.075 ms
/ # nslookup web
Server:    10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name:      web
Address 1: 10.0.0.213 web.default.svc.cluster.local
/ # ping web
PING web (10.0.0.213): 56 data bytes
64 bytes from 10.0.0.213: seq=0 ttl=64 time=0.034 ms

/ # nslookup kubernetes
Server:    10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local

# 程序访问service 就可以写service name

K8S 高可用介绍

LB : Nginx LVS HaProxy
4层负载  端口转发 stream    TCP 性能更好
7层负载  业务转发 HTTP 应用层协议满足更多数据转发

高可用软件: keepalived 创建vip 健康检查和故障转移 用户访问vip
node 连接 vip

部署master2

172.17.70.245 k8s-master2
172.17.70.246 k8s-master
172.17.70.247 k8s-node1
172.17.70.248 k8s-node2

# 将kubernetes 传过去
[root@k8s-master ~]# scp -r /opt/kubernetes root@172.17.70.245:/opt

# 管理服务的service文件
[root@k8s-master ~]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@172.17.70.245:/usr/lib/systemd/system

# 拷贝etcd证书目录
[root@k8s-master2 ssl]# mkdir -p /opt/etcd
[root@k8s-master ~]# scp -r /opt/etcd/ssl root@172.17.70.245:/opt/etcd

# 修改配置文件
# 修改apiserver配置文件为本地IP：
[root@k8s-master2 cfg]# cat kube-apiserver.conf | grep 245
--bind-address=172.17.70.245 \
--advertise-address=172.17.70.245 \

1 2	# 传递 kubectl [root@k8s-master ~]# scp /usr/local/bin/kubectl root@172.17.70.245:/usr/bin/

# 启动服务
[root@k8s-node1 cfg]# systemctl daemon-reload
systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
systemctl enable kube-apiserver
systemctl enable kube-controller-manager
systemctl enable kube-scheduler

# 查看资源 有数据即为正常
[root@k8s-master2 ssl]# kubectl get node
NAME        STATUS   ROLES    AGE    VERSION
k8s-node1   Ready    <none>   109m   v1.16.0
k8s-node2   Ready    <none>   107m   v1.16.0

[root@k8s-master2 ssl]# kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
busybox                1/1     Running   0          55m
web-866f97c649-mksh6   1/1     Running   0          80m

部署 Nginx 负载均衡

172.17.70.245 k8s-master2
172.17.70.246 k8s-master
172.17.70.247 k8s-node1
172.17.70.248 k8s-node2
172.17.70.249 nginx-master
172.17.70.250 nginx-backup

# 下载rpm包
nginx rpm包：http://nginx.org/packages/rhel/7/x86_64/RPMS/
[root@k8s-master tmp]# wget http://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.16.1-1.el7.ngx.x86_64.rpm
[root@k8s-master tmp]# scp nginx-1.16.1-1.el7.ngx.x86_64.rpm root@172.17.70.249:/root
[root@k8s-master tmp]# scp nginx-1.16.1-1.el7.ngx.x86_64.rpm root@172.17.70.250:/root
# 安装
[root@nginx-master ~]# rpm -ivh nginx-1.16.1-1.el7.ngx.x86_64.rpm

# 两台nginx 都要完成
# 修改配置文件 增加4层端口搭理到master
[root@nginx-backup ~]# vi /etc/nginx/nginx.conf 

stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
                server 172.17.70.246:6443;
                server 172.17.70.245:6443;
            }

    server {
       listen 6443;
       proxy_pass k8s-apiserver;
    }
}

# 启动

nginx -t
systemctl start nginx
systemctl enable nginx
systemctl status nginx

安装 keepalived 心跳检测

1	[root@nginx-master ~]# yum install keepalived -y

# 编辑配置文件
[root@nginx-master keepalived]# cd /etc/keepalived/

[root@nginx-master keepalived]# mv keepalived.conf keepalived.conf_bak

[root@nginx-master keepalived]# vim keepalived.conf


global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
    priority 100    # 优先级，备服务器设置 90
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.17.70.252/24 # vip
    }
    track_script {
        check_nginx
    }
}

# 检查脚本 
# keepalived 对 脚本返回状态码做判断 0 和 1 0为正常

[root@nginx-master ~]# vim check_nginx.sh 

#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    exit 1
else
    exit 0
fi


[root@nginx-backup keepalived]# cp /root/check_nginx.sh .
[root@nginx-backup keepalived]# chmod +x check_nginx.sh

# 备机配置文件
[root@nginx-backup keepalived]# vim keepalived.conf



global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_BACKUP
}

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
    priority 90    # 优先级，备服务器设置 90
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.17.70.252/24
    }
    track_script {
        check_nginx
    }
}

# 启动keepalived 
systemctl start keepalived
systemctl enable keepalived
systemctl status keepalived

[root@nginx-master keepalived]# ssh root@172.17.70.252

# 阿里云还是用自己的负载均衡吧
# 关闭nginx测试是否偏移到备节点。

修改Node连接VIP

# 将Node连接VIP：
 
# cd /opt/kubernetes/cfg
# grep 172 *
bootstrap.kubeconfig:    server: https://172.17.70.246:6443
kubelet.kubeconfig:    server: https://172.17.70.246:6443
kube-proxy.kubeconfig:    server: https://172.17.70.246:6443

# 批量修改：改成vip地址
sed -i 's#172.17.70.246#172.17.70.252#g' *

# 重启node服务 
systemctl restart kubelet
systemctl restart kube-proxy

# 看看nginx有没有新日志请求
tailf /var/log/nginx/k8s-access.log 
# 总共四个组件进来 四次请求

# 再任意主节点测试 虽然是本地测试
kubectl get node 

# 找其他公网机器
cat /opt/kubernetes/cfg/token.csv 
curl -k --header "Authorization: Bearer c47ffb939f5ca36231d9e3121a252940" https://172.17.70.252:6443/version


# 单master 测试
[root@nginx-backup ~]# curl -k --header "Authorization: Bearer c47ffb939f5ca36231d9e3121a252940" https://172.17.70.249:6443/version
{
  "major": "1",
  "minor": "16",
  "gitVersion": "v1.16.0",
  "gitCommit": "2bd9643cee5b3b3a5ecbd3af49d09018f0773c77",
  "gitTreeState": "clean",
  "buildDate": "2019-09-18T14:27:17Z",
  "goVersion": "go1.12.9",
  "compiler": "gc",
  "platform": "linux/amd64"


# 可以关闭一个nginx
systemctl stop nginx
# 查看 vip是否便宜
# 最后再查看是否可以正常访问
curl -k --header "Authorization: Bearer c47ffb939f5ca36231d9e3121a252940" https://172.17.70.252:6443/version
# 可以的话 就为正常 任意一个master无法访问 也可以访问到另外一台 nginx挂了 也可以通过另外的nginx负载访问